emacs rendering comparisson between emacs23 and emacs26.3

emacs rendering comparisson between emacs23 and emacs26.3

[rrandresf]

Hi.

On my TODO list (kind of worn hole). I have several notes.

One of them is comparing the rendering on emacs23 and emacs26.3. Today
Alan Mackenzie share an elisp Snippet:

--8<---------------cut here---------------start------------->8---
(defmacro time-it (&rest forms)
  "Time the running of a sequence of forms using `float-time'.
Call like this: \"M-: (time-it (foo ...) (bar ...) ...)\"."
  `(let ((start (float-time)))
    ,@forms
    (- (float-time) start)))

(defun time-scroll (&optional arg)
  (interactive "P")
  (message "%s"
           (time-it
            (condition-case nil
                (while t
                  (if arg (scroll-down) (scroll-up))
                  (sit-for 0))
              (error nil)))))
--8<---------------cut here---------------end--------------->8---

Alan suggested this to Andrea Corallo (BTW nice work with your branch
Andrea. and I still remember the discussion about numbering and source
code Alan)
--8<---------------cut here---------------start------------->8---
, visit .../emacs/src/xdisp.c, and do M-: (time-scroll)
--8<---------------cut here---------------end--------------->8---

When I saw this. The first thing that come to my mind was (it is
useful on one of the items of my TODO'S). So I did It on xdisp.c from emacs26-3 source
code.

Applying it on emacs23 and on emacs26 (on an opi+2e machine with 1 Ghz
processor)
This is the result:
--8<---------------cut here---------------start------------->8---
emacs-23
162.7052059173584
emacs-26.3
281.4721345901489
--8<---------------cut here---------------end--------------->8---


The difference is very notorious. I realized it from a long time on
another ARM machine with 480 Mhz (which i still use aka my phone). But
I have not got the time and tools for letting you guys know about it. So
thanks Alan for sharing it.

Teoretically. What would be the cause of this difference on rendering
between versions?

Best Regards

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> --8<---------------cut here---------------start------------->8---
> emacs-23
> 162.7052059173584
> emacs-26.3
> 281.4721345901489
> --8<---------------cut here---------------end--------------->8---
[...]
> Teoretically. What would be the cause of this difference on rendering
> between versions?

It's hard to tell without digging deeper, it could be in part due to
changes to the actual redisplay code or to the rest of the C code, but
I suspect it's mostly due to improvements (featurewise) in CC-mode.

I just compared this test's performance using `c-mode` vs using
`sm-c-mode` (a comparatively very bare-bones major mode limited to
"normal" C (no support for any C++ feature nor for old K&R-style C, nor
for very modern C)), and the speed difference was even larger.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: rrandresf@gmail.com
> Date: Sat, 21 Mar 2020 22:19:27 +0000
> 
> Teoretically. What would be the cause of this difference on rendering
> between versions?

Emacs 23 didn't support bidirectional display.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello!

On Sat, Mar 21, 2020 at 22:19:27 +0000, rrandresf@gmail.com wrote:
> Hi.

> On my TODO list (kind of worn hole). I have several notes.

> One of them is comparing the rendering on emacs23 and emacs26.3. Today
> Alan Mackenzie share an elisp Snippet:

[ .... ]

> When I saw this. The first thing that come to my mind was (it is
> useful on one of the items of my TODO'S). So I did It on xdisp.c from
> emacs26-3 source code.

> Applying it on emacs23 and on emacs26 (on an opi+2e machine with 1 Ghz
> processor)
> This is the result:
> --8<---------------cut here---------------start------------->8---
> emacs-23
> 162.7052059173584
> emacs-26.3
> 281.4721345901489
> --8<---------------cut here---------------end--------------->8---


> The difference is very notorious. I realized it from a long time on
> another ARM machine with 480 Mhz (which i still use aka my phone). But
> I have not got the time and tools for letting you guys know about it. So
> thanks Alan for sharing it.

> Teoretically. What would be the cause of this difference on rendering
> between versions?

There have been lots of bug fixes and new features in CC Mode.  Many of
these have needed extra processing.

Still, an increase of ~75% in processing time over the best part of a
decade isn't that bad.  Surely?  Certainly not when compared with the
increase in available computing power.

One workaround is to reduce the fontification level with
font-lock-maximum-decoration.  A value of 3 is full fontification, 2 is
quite a bit less.

> Best Regards

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Amin Bandali]

[-- Attachment #1: Type: text/plain, Size: 726 bytes --]

Hello,

I may be in the minority, but...

Alan Mackenzie <acm@muc.de> writes:

[...]
>
> Still, an increase of ~75% in processing time over the best part of a
> decade isn't that bad.  Surely?  Certainly not when compared with the
> increase in available computing power.
>
[...]

Just as a data point, my main machine is a ThinkPad released in 2008
with a Core 2 Duo processor, which is not nearly as fast as the newer
Core i5/i7 processors found in more recent machines since.

That is not to say I don't enjoy and appreciate useful new features in
GNU Emacs, but rather that I think we should keep a close eye on the
performance penalties accumulated over time and (continue) trying to
minimize them as much as we can. :-)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Amin Bandali <bandali@gnu.org>
> Date: Sun, 22 Mar 2020 11:08:25 -0400
> 
> That is not to say I don't enjoy and appreciate useful new features in
> GNU Emacs, but rather that I think we should keep a close eye on the
> performance penalties accumulated over time and (continue) trying to
> minimize them as much as we can. :-)

Indeed, we should, and we do.  Several slowdowns in CC mode were
reported during the recent years, and at least some of them were
fixed.  Feel free to report more, preferably with profile data to show
the bottlenecks and test cases to use in making CC mode faster.

Thanks.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Still, an increase of ~75% in processing time over the best part of a
  > > decade isn't that bad.  Surely?  Certainly not when compared with the
  > > increase in available computing power.

There has been no increase in computing power in the Free World.
I expect that Amin chooses these older machines

  > Just as a data point, my main machine is a ThinkPad released in 2008
  > with a Core 2 Duo processor, which is not nearly as fast as the newer
  > Core i5/i7 processors found in more recent machines since.

for the same reason I choose them: the FSF recommends them, because newer
processors have back doors.  Intel processors newer than that contain
the "Management Engine" back door and can't start up without it.

So please, everyone, take care to maintain good performance on these
older machines.

Is it possible, to restore most of the lost peformance by deactivating
some new features?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Richard.

On Sun, Mar 22, 2020 at 22:51:39 -0400, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

>   > > Still, an increase of ~75% in processing time over the best part of a
>   > > decade isn't that bad.  Surely?  Certainly not when compared with the
>   > > increase in available computing power.

> There has been no increase in computing power in the Free World.
> I expect that Amin chooses these older machines

>   > Just as a data point, my main machine is a ThinkPad released in 2008
>   > with a Core 2 Duo processor, which is not nearly as fast as the newer
>   > Core i5/i7 processors found in more recent machines since.

> for the same reason I choose them: the FSF recommends them, because newer
> processors have back doors.  Intel processors newer than that contain
> the "Management Engine" back door and can't start up without it.

That's a good point.  Presumably processors without these back doors are
difficult to come by nowadays.

> So please, everyone, take care to maintain good performance on these
> older machines.

> Is it possible, to restore most of the lost peformance by deactivating
> some new features?

The only way at the moment (for CC Mode) is to change
font-lock-maximum-decoration from t to 2.

Most of the increased processing comes from fixing bugs rather than
adding new features.  Keeping up with, in particular, the latest C++
standard (which seems to appear every three years) is a challenge, as
they steadily make shallow syntactic analysis more and more difficult.

> -- 
> Dr Richard Stallman
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> Most of the increased processing comes from fixing bugs rather than
> adding new features.  Keeping up with, in particular, the latest C++
> standard (which seems to appear every three years) is a challenge, as
> they steadily make shallow syntactic analysis more and more difficult.

FWIW: Not just *shallow*!


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[rrandresf]

Hi Alan. Hi. Richard.

>> contain the "Management Engine" back door and can't start up
>> without it.

Alan> That's a good point.  Presumably processors without these back
Alan> doors are difficult to come by nowadays.

>> Is it possible, to restore most of the lost peformance by
>> deactivating some new features?

Alan> The only way at the moment (for CC Mode) is to change
Alan> font-lock-maximum-decoration from t to 2.

BTW. On my comparissons that variable was set no nil
as Doc says:
--8<---------------cut here---------------start------------->8---
If nil, use the default decoration (typically the minimum available).
--8<---------------cut here---------------end--------------->8---
Also xdisp.c. Just load c-mode not c++-mode

Alan> Most of the increased processing comes from fixing bugs rather
Alan> than adding new features.  Keeping up with, in particular, the
Alan> latest C++ standard (which seems to appear every three years) is
Alan> a challenge, as they steadily make shallow syntactic analysis
Alan> more and more difficult.

I have complete the same test on all Emacsen (Sxemacs-git and Xemacs
21.5). Nothing beats GNU/Emacs23 on rendering.

As Eli pointed out on another email on this same thread. It could be
BIDI. I would need to check on configure If it is possible to deactivate
BIDI support and see what difference it Does.

With the help of Michael Albinus I have update the emacs23 builtin tramp
version to the latest one that supports GNU/Emacs23 (wellcome multihops and
adb). I think perhaps in the future I could upgrade org-mode to the last
version that supports GNU/Emacs23 (¿?). And I am going to stay with that setup on
low power machines. BTW. Not having magit on GNU/Emacs23 have made me
discover the niceties about vc-dir (sometimes it hangs on GNU/Emacs23) not
on 26.3.

Also on the same subject. There are several packages that require
GNU/Emacs24 as minimum but they DO work fine with GNU/Emacs23. Yes I Do use
GNU/Emacs23 with package.el. and GNU/Emacs23 coexist with GNU/Emacs26.3

Also GNU/Emacs23. with all the patches from Gentoo (thanks Ulrich
Mueller). compiles fine with latest and greatest tooling {glibc 2.30} on x86 (32 and 64 bits). But does not compile on
ARM (a low power machine like orange pi). It stays on a loop. I have no have the time for attaching gdb to
it. But I plan on  doing it sometime in the future. So for ARM architecture just
CANNOT_DUMP works with a lot of tweaking for compilation (Do not try it).

Best Regards

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: rrandresf@gmail.com
> Date: Thu, 26 Mar 2020 20:38:22 +0000
> Cc: Richard Stallman <rms@gnu.org>, emacs-devel@gnu.org
> 
> As Eli pointed out on another email on this same thread. It could be
> BIDI. I would need to check on configure If it is possible to deactivate
> BIDI support and see what difference it Does.

It's not possible.  Bidirectional editing support in Emacs is
unconditional, it isn't an optional feature.

What I would suggest is to compare with Emacs 24.1, which already
supported bidi.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[andrés ramírez]

Hi Eli.

Eli> It's not possible.  Bidirectional editing support in Emacs is
Eli> unconditional, it isn't an optional feature.
Mmmm.

Eli> What I would suggest is to compare with Emacs 24.1, which already
Eli> supported bidi.

Well. Based on past experience.
I am going to try to compile emacs24.1 on this ARM machine. If not
possible. I am going to move to another machine (compaq n600c an x86
with Ghz processor and 256 Mb memory). On of this two options should work.

Then I am going to try the test with default cc-mode (the one that comes
with 24.1). And after that I
am going to plug cc-mode from 23.4 on 24.1. Just for taking out cc-mode
out of equation and concentrating just on bidi.

Best Regards

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

To tell the people with the strongest commitment to freedom to use an
old version of Emacs is to choose failure.  We need to re-optimize the
Emacs master so it gives good performance on those machines.

Suggesting that we disable certain features is ok.  Perhaps there
could be a command to select several of these modes together.

  > The only way at the moment (for CC Mode) is to change
  > font-lock-maximum-decoration from t to 2.

That is ok as part of the solution.

Regarding C++ indentation, I have two suggestions.

* We could support a faster indentation mode for C code
and for C++ files that use a limited repertoire of C++ constructs.

* Maybe we can come up with additional tricks comparable to
{-in-column-0 to optimize parsing.  Emacs users would put these things
into source files so as to get better peformance.

  > BTW. On my comparissons that variable was set no nil
  > as Doc says:
  > --8<---------------cut here---------------start------------->8---
  > If nil, use the default decoration (typically the minimum available).
  > --8<---------------cut here---------------end--------------->8---
  > Also xdisp.c. Just load c-mode not c++-mode

What were the results of the comparisons you did with this setting?
(I wasn't following this topic until yesterday.)

  > As Eli pointed out on another email on this same thread. It could be
  > BIDI. I would need to check on configure If it is possible to deactivate
  > BIDI support and see what difference it Does.

Would you please try that?  But don't limit yourself to existing
configuration options -- try changing it at the C level if you can.
If you find that this makes a big difference (a "bi di" ?), we can
arrange to turn it on and off at run time.

  > With the help of Michael Albinus I have update the emacs23 builtin tramp
  > version to the latest one that supports GNU/Emacs23

The slash is not called for here.  In "GNU Emacs", "GNU" acts as an adjectival
modifying "Emacs", so it is correct to separate them with just a space.

At least that is one problem that is easy to fix.

							(wellcome multihops and
  > adb).

Has Tramp become a lot slower?  Or is it only that the current Tramp doesn't
work on Emacs 23?

	  I think perhaps in the future I could upgrade org-mode to the last
  > version that supports GNU Emacs 23 (¿?). And I am going to stay with that setup on
  > low power machines.

Has Org mode become a lot slower?  Or is it only that the current Org
mode doesn't work on Emacs 23?


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It's not possible.  Bidirectional editing support in Emacs is
  > unconditional, it isn't an optional feature.

It isn't optional _now_ but maybe there is an opportunity to optimize
redisplay if the userssays "no BIDI".

Maybe some of the pre-bidi code could be brought back and used
when the user says there is no BIDI in the buffer.

  > What I would suggest is to compare with Emacs 24.1, which already
  > supported bidi.

It would be interesting to compare 24 and 23, to see what slowdown
there is.  And compare other subsequent versions to see what slowdown
there is.  These could give hints for what operations are worth trying
to optimize.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: rrandresf@gmail.com, acm@muc.de, emacs-devel@gnu.org
> Date: Fri, 27 Mar 2020 22:48:38 -0400
> 
>   > It's not possible.  Bidirectional editing support in Emacs is
>   > unconditional, it isn't an optional feature.
> 
> It isn't optional _now_ but maybe there is an opportunity to optimize
> redisplay if the userssays "no BIDI".
> 
> Maybe some of the pre-bidi code could be brought back and used
> when the user says there is no BIDI in the buffer.

If you find volunteers to work on such changes, sure, why not.  I
don't volunteer, mind you: the display code is complicated enough even
without that.

> It would be interesting to compare 24 and 23, to see what slowdown
> there is.

I expect an average twofold slowdown due to bidirectional support.
The OP indicates less than that (comparing Emacs 23 with Emacs 26), so
it's reasonable, IMO, and probably means the CC Mode contribution is
relatively minor.

The tricky part is testing the bidirectional part and the CC Mode part
separately, by loading the old Emacs-23 CC Mode files into later
versions of Emacs.  Maybe Alan could help by providing a Lisp file
which, when loaded, would load CC Mode from a given directory -- this
should be run before visiting any C/C++ file.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Date: Sat, 28 Mar 2020 10:09:59 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org
> 
> > It would be interesting to compare 24 and 23, to see what slowdown
> > there is.
> 
> I expect an average twofold slowdown due to bidirectional support.

Actually, I take that back: in benchmarks such as scroll-up-benchmark,
and in a buffer with predominantly ASCII text, the effect of
bidirectional support should be very small, perhaps 10% or 20%.

So the issue of testing old v23.x CC Mode with newer versions of Emacs
becomes more important than I originally thought.  Just as a teaser,
here's the profile I obtained by scrolling in Emacs 26.3 through the
entire xdisp.c, it seems to point to c-find-decl-spots and its
immediate callers and subroutines as the hot spots:

- command-execute                                                1446 100%
 - call-interactively                                            1446 100%
  - funcall-interactively                                        1446 100%
   - execute-extended-command                                    1446 100%
    - command-execute                                            1446 100%
     - call-interactively                                        1446 100%
      - funcall-interactively                                    1446 100%
       - scroll-up-benchmark                                     1446 100%
        - let                                                    1446 100%
         - condition-case                                        1446 100%
          - while                                                1446 100%
           - redisplay                                           1351  93%
            - redisplay_internal (C function)                    1071  74%
             - jit-lock-function                                 1064  73%
              - jit-lock-fontify-now                             1063  73%
               - jit-lock--run-functions                         1063  73%
                - run-hook-wrapped                               1063  73%
                 - #<compiled 0x2000000008356520>                1063  73%
                  - font-lock-fontify-region                     1061  73%
                   - c-font-lock-fontify-region                  1061  73%
                    - font-lock-default-fontify-region            982  67%
                     - font-lock-fontify-keywords-region          969  67%
                      - c-font-lock-declarations                  686  47%
                       - c-find-decl-spots                        674  46%
                        - #<compiled 0x20000000077e07c0>          477  32%
                         - c-get-fontification-context            253  17%
                          - c-looking-at-or-maybe-in-bracelist    203  14%
                           - c-backward-token-2                   196  13%
                            - c-backward-sws                       35   2%
                             - c-beginning-of-macro                12   0%
                                back-to-indentation                 6   0%
                          - c-backward-token-2                     16   1%
                           - c-backward-sws                         7   0%
                              c-beginning-of-current-token          1   0%
                             c-beginning-of-current-token           1   0%
                          - c-back-over-compound-identifier        15   1%
                           - c-backward-sws                         6   0%
                            - c-beginning-of-macro                  3   0%
                               back-to-indentation                  3   0%
                             c-on-identifier                        5   0%
                             c-backward-token-2                     2   0%
                          - c-backward-sws                          6   0%
                             c-beginning-of-macro                   1   0%
                         - c-forward-decl-or-cast-1               125   8%
                          - c-forward-type                         65   4%
                           - c-forward-name                        18   1%
                              c-forward-sws                         4   0%
                           - c-syntactic-content                    4   0%
                              c-forward-sws                         1   0%
                              apply                                 1   0%
                             c-add-type                             3   0%
                           - c-forward-keyword-clause               2   0%
                              c-forward-sws                         1   0%
                          - c-backward-token-2                      9   0%
                             c-backward-sws                         3   0%
                             c-beginning-of-current-token           2   0%
                          - c-directly-in-class-called-p            3   0%
                           - c-looking-at-decl-block                2   0%
                            - c-syntactic-skip-backward             2   0%
                             - c-parse-state                        2   0%
                              - c-parse-state-1                     1   0%
                               - c-parse-state-get-strategy         1   0%
                                - c-beginning-of-macro              1   0%
                                   back-to-indentation              1   0%
                              - c-beginning-of-macro                1   0%
                                 back-to-indentation                1   0%
                           - c-parse-state                          1   0%
                              c-parse-state-1                       1   0%
                          - c-forward-name                          3   0%
                             c-forward-sws                          2   0%
                            c-forward-sws                           3   0%
                            c-forward-annotation                    1   0%
                         - c-forward-label                         33   2%
                          - c-after-conditional                     8   0%
                           - c-backward-token-2                     6   0%
                            - c-backward-sws                        5   0%
                             - c-beginning-of-macro                 1   0%
                                back-to-indentation                 1   0%
                               c-beginning-of-current-token         1   0%
                          - c-syntactic-re-search-forward           7   0%
                           - c-beginning-of-macro                   4   0%
                              back-to-indentation                   1   0%
                            c-forward-sws                           4   0%
                            c-beginning-of-current-token            1   0%
                         - c-backward-sws                          12   0%
                          - c-beginning-of-macro                    2   0%
                             back-to-indentation                    1   0%
                         - c-font-lock-single-decl                 10   0%
                          - c-font-lock-declarators                 9   0%
                           - c-forward-declarator                   8   0%
                            - c-backward-sws                        3   0%
                               c-beginning-of-current-token         2   0%
                            - c-forward-name                        2   0%
                               c-forward-sws                        1   0%
                              c-syntactic-re-search-forward         1   0%
                         - c-backward-token-2                      10   0%
                            c-backward-sws                          4   0%
                            c-beginning-of-current-token            3   0%
                        - c-bs-at-toplevel-p                      141   9%
                         - c-brace-stack-at                       135   9%
                          - c-update-brace-stack                  126   8%
                           - c-syntactic-re-search-forward         87   6%
                            - c-beginning-of-macro                 22   1%
                               back-to-indentation                 15   1%
                             c-beginning-of-current-token          17   1%
                             match-string-no-properties             1   0%
                        - c-beginning-of-macro                     13   0%
                           back-to-indentation                      9   0%
                          c-forward-sws                            11   0%
                          c-backward-sws                            1   0%
                      - c-font-lock-cut-off-declarators           116   8%
                       - c-back-over-member-initializers           35   2%
                        - c-parse-state                             8   0%
                         - c-parse-state-1                          8   0%
                          - c-remove-stale-state-cache              3   0%
                           - c-beginning-of-macro                   3   0%
                              back-to-indentation                   1   0%
                          - c-append-to-state-cache                 2   0%
                           - c-beginning-of-macro                   2   0%
                              back-to-indentation                   2   0%
                          - c-parse-state-get-strategy              1   0%
                             c-beginning-of-macro                   1   0%
                        - c-just-after-func-arglist-p               7   0%
                           c-beginning-of-statement-1               7   0%
                        - c-backward-sws                            5   0%
                         - c-beginning-of-macro                     2   0%
                            back-to-indentation                     2   0%
                        - c-back-over-compound-identifier           2   0%
                           c-on-identifier                          2   0%
                       - c-syntactic-skip-backward                 30   2%
                        - c-parse-state                            16   1%
                         - c-parse-state-1                         16   1%
                          - c-append-lower-brace-pair-to-state-cache   5   0%
                             c-beginning-of-macro                   1   0%
                            c-remove-stale-state-cache              2   0%
                            c-append-to-state-cache                 2   0%
                          - c-remove-stale-state-cache-backwards    2   0%
                             c-state-balance-parens-backwards       1   0%
                           - c-state-literal-at                     1   0%
                              c-state-pp-to-literal                 1   0%
                          - c-parse-state-get-strategy              1   0%
                           - c-beginning-of-macro                   1   0%
                              #<compiled 0x2000000001a3f060>        1   0%
                          c-beginning-of-macro                      1   0%
                       - c-determine-limit                         22   1%
                          c-state-semi-pp-to-literal                8   0%
                        - c-determine-limit                         3   0%
                           c-state-semi-pp-to-literal               2   0%
                       - c-parse-state                              6   0%
                        - c-parse-state-1                           6   0%
                           c-append-lower-brace-pair-to-state-cache 6   0%
                       - c-forward-decl-or-cast-1                   5   0%
                          c-forward-annotation                      1   0%
                        - c-forward-type                            1   0%
                           c-forward-name                           1   0%
                       - c-get-fontification-context                3   0%
                        - c-looking-at-or-maybe-in-bracelist        3   0%
                           c-backward-token-2                       3   0%
                       - c-at-toplevel-p                            3   0%
                        - c-search-uplist-for-classkey              3   0%
                         - c-looking-at-decl-block                  3   0%
                          - c-syntactic-skip-backward               3   0%
                           - c-parse-state                          2   0%
                            - c-parse-state-1                       1   0%
                               c-remove-stale-state-cache-backwards 1   0%
                              c-beginning-of-macro                  1   0%
                         c-forward-sws                              2   0%
                       - c-font-lock-single-decl                    1   0%
                          c-font-lock-declarators                   1   0%
                      - c-font-lock-enclosing-decls                54   3%
                       - c-determine-limit                         25   1%
                        - c-state-semi-pp-to-literal               12   0%
                           #<compiled 0x2000000001a640b8>           1   0%
                        - c-determine-limit                         4   0%
                           c-state-semi-pp-to-literal               3   0%
                       - c-syntactic-skip-backward                 16   1%
                        - c-parse-state                            10   0%
                         - c-parse-state-1                          9   0%
                          - c-parse-state-get-strategy              3   0%
                           - c-beginning-of-macro                   2   0%
                              back-to-indentation                   1   0%
                          - c-remove-stale-state-cache-backwards    3   0%
                             c-state-balance-parens-backwards       1   0%
                           - c-state-literal-at                     1   0%
                              c-state-pp-to-literal                 1   0%
                             c-beginning-of-macro                   1   0%
                          - c-append-to-state-cache                 1   0%
                             c-beginning-of-macro                   1   0%
                           c-beginning-of-macro                     1   0%
                        - c-backward-sws                            2   0%
                         - c-beginning-of-macro                     1   0%
                            back-to-indentation                     1   0%
                          copy-tree                                 1   0%
                          c-beginning-of-macro                      1   0%
                       - c-parse-state                              7   0%
                        - c-parse-state-1                           6   0%
                         - c-append-to-state-cache                  4   0%
                            c-beginning-of-macro                    1   0%
                           c-remove-stale-state-cache               1   0%
                        - c-beginning-of-macro                      1   0%
                           back-to-indentation                      1   0%
                         c-forward-sws                              1   0%
                      - c-font-lock-complex-decl-prepare           25   1%
                       - c-parse-state                             15   1%
                        - c-parse-state-1                          15   1%
                         - c-append-to-state-cache                  8   0%
                            c-append-lower-brace-pair-to-state-cache  1   0%
                         - c-parse-state-get-strategy               3   0%
                          - c-beginning-of-macro                    2   0%
                             back-to-indentation                    2   0%
                          - c-get-fallback-scan-pos                 1   0%
                           - beginning-of-defun                     1   0%
                              beginning-of-defun-raw                1   0%
                         - c-remove-stale-state-cache               2   0%
                          - c-beginning-of-macro                    2   0%
                             back-to-indentation                    1   0%
                       - c-backward-sws                             7   0%
                        - c-beginning-of-macro                      5   0%
                           back-to-indentation                      2   0%
                        #<compiled 0x20000000077e7950>             13   0%
                        #<compiled 0x20000000077e7538>             12   0%
                      - c-font-lock-enum-tail                      11   0%
                       - c-backward-over-enum-header                6   0%
                        - c-backward-token-2                        4   0%
                         - c-backward-sws                           4   0%
                            c-beginning-of-current-token            1   0%
                       - c-parse-state                              5   0%
                        - c-parse-state-1                           4   0%
                         - c-remove-stale-state-cache               2   0%
                            c-beginning-of-macro                    1   0%
                           c-append-to-state-cache                  1   0%
                          c-beginning-of-macro                      1   0%
                        #<compiled 0x20000000077e79d0>             11   0%
                        #<compiled 0x20000000077e7aa0>              7   0%
                      - c-font-lock-invalid-single-quotes           7   0%
                       - c-literal-limits                           3   0%
                          c-state-full-pp-to-literal                3   0%
                        #<compiled 0x20000000077e7678>              7   0%
                      - #<compiled 0x2000000007807748>              3   0%
                       - c-font-lock-doc-comments                   3   0%
                        - c-literal-start                           2   0%
                           c-state-semi-pp-to-literal               2   0%
                        #<compiled 0x20000000077e7808>              2   0%
                        c-font-lock-enum-body                       1   0%
                     - font-lock-fontify-syntactically-region       9   0%
                        syntax-ppss                                 7   0%
                    - c-before-context-fl-expand-region            77   5%
                     - mapc                                        77   5%
                      - #<compiled 0x20000000078876d8>             77   5%
                       - c-context-expand-fl-region                76   5%
                        - c-fl-decl-start                          62   4%
                         - c-syntactic-skip-backward               27   1%
                          - c-parse-state                          11   0%
                           - c-parse-state-1                       11   0%
                            - c-append-to-state-cache               5   0%
                             - c-beginning-of-macro                 1   0%
                                back-to-indentation                 1   0%
                               c-append-lower-brace-pair-to-state-cache   1   0%
                            - c-parse-state-get-strategy            1   0%
                             - c-beginning-of-macro                 1   0%
                                back-to-indentation                 1   0%
                              c-remove-stale-state-cache            1   0%
                          - c-beginning-of-macro                    1   0%
                             back-to-indentation                    1   0%
                         - c-determine-limit                       17   1%
                            c-state-semi-pp-to-literal              9   0%
                          - c-determine-limit                       2   0%
                           - c-determine-limit                      1   0%
                              c-state-semi-pp-to-literal            1   0%
                             c-state-semi-pp-to-literal             1   0%
                         - c-literal-start                          6   0%
                          - c-state-semi-pp-to-literal              6   0%
                             c-parse-ps-state-below                 2   0%
                         - c-looking-at-or-maybe-in-bracelist       5   0%
                            c-backward-token-2                      5   0%
                         - c-forward-type                           3   0%
                            c-forward-name                          2   0%
                           c-backward-sws                           1   0%
                        - c-fl-decl-end                            11   0%
                         - c-literal-start                          7   0%
                          - c-state-semi-pp-to-literal              6   0%
                             c-parse-ps-state-below                 1   0%
                           c-on-identifier                          1   0%
                           c-forward-declarator                     1   0%
                         - c-backward-sws                           1   0%
                            c-beginning-of-macro                    1   0%
                    bug-reference-fontify                           2   0%
               file-remote-p                                        3   0%
             - find-image                                           2   0%
                image-search-load-path                              2   0%
             - eval                                                 1   0%
                if                                                  1   0%
            - timer-event-handler                                   1   0%
             - apply                                                1   0%
                jit-lock-force-redisplay                            1   0%

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Michael Albinus]

Richard Stallman <rms@gnu.org> writes:

> Has Tramp become a lot slower?  Or is it only that the current Tramp doesn't
> work on Emacs 23?

The latter. Current Tramp (2.5.x) supports Emacs 25 pp.

Best regards, Michael.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Fri, 27 Mar 2020 22:40:47 -0400
> Cc: acm@muc.de, emacs-devel@gnu.org
> 
> To tell the people with the strongest commitment to freedom to use an
> old version of Emacs is to choose failure.  We need to re-optimize the
> Emacs master so it gives good performance on those machines.

That is of course the ideal we should always strive to, but it's
unreasonable to expect newer versions of Emacs to stay as fast as the
oder ones: we do add functionality, and some of that eats up CPU
cycles.  We do try to optimize performance, and occasionally do make
newer versions faster (e.g., Emacs 26 is about 30% faster than Emacs
25 on my machine, when scrolling through a large C file), but those
are exceptions rather than the rule, because significant speedups are
only possible by redesign and reimplementation of core functions,
something we do rarely.

> Suggesting that we disable certain features is ok.

Then my suggestion is to disable font-lock: that makes scrolling
through xdisp.c in Emacs 26.3 8 times faster, and almost as fast as
Emacs 23.4.

> Regarding C++ indentation, I have two suggestions.
> 
> * We could support a faster indentation mode for C code
> and for C++ files that use a limited repertoire of C++ constructs.
> 
> * Maybe we can come up with additional tricks comparable to
> {-in-column-0 to optimize parsing.  Emacs users would put these things
> into source files so as to get better peformance.

I think the way to increase performance of C/C++ fontification,
indentation, etc. is to use syntactic analysis in C (or another
similarly fast compiled language).  Several programs/libraries are
available out there for this purpose, and we should encourage
interested individuals to work on making these used by our major modes
that support programming languages.  The time we could do that with
ad-hoc regexps and the likes of syntax-ppss has gone, IMO.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Date: Sat, 28 Mar 2020 12:58:00 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org
> 
> That is of course the ideal we should always strive to, but it's
> unreasonable to expect newer versions of Emacs to stay as fast as the
> oder ones: we do add functionality, and some of that eats up CPU
  ^^^^
"older", sorry.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Ulrich Mueller]

>>>>> On Fri, 27 Mar 2020, Richard Stallman wrote:

>> It's not possible.  Bidirectional editing support in Emacs is
>> unconditional, it isn't an optional feature.

> It isn't optional _now_ but maybe there is an opportunity to optimize
> redisplay if the userssays "no BIDI".

> Maybe some of the pre-bidi code could be brought back and used
> when the user says there is no BIDI in the buffer.

>> What I would suggest is to compare with Emacs 24.1, which already
>> supported bidi.

> It would be interesting to compare 24 and 23, to see what slowdown
> there is.  And compare other subsequent versions to see what slowdown
> there is.  These could give hints for what operations are worth trying
> to optimize.

I've run the benchmark for several Emacs versions, on a Thinkpad L580
from 2018 (Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz). In all cases, I
used xdisp.c from version 26.3 as the test file.

  18.59        n/a    0.320
  23.4       5.436    1.019
  24.5      25.482    1.636
  25.3      22.370    1.610
  26.3      16.533    1.911
  27.0.90   17.602    2.339

First column is the version, second and third column are the times with
font-lock-mode enabled and disabled, respectively.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Eli.

On Sat, Mar 28, 2020 at 10:09:59 +0300, Eli Zaretskii wrote:
> > From: Richard Stallman <rms@gnu.org>
> > Cc: rrandresf@gmail.com, acm@muc.de, emacs-devel@gnu.org
> > Date: Fri, 27 Mar 2020 22:48:38 -0400

[ .... ]

> I expect an average twofold slowdown due to bidirectional support.
> The OP indicates less than that (comparing Emacs 23 with Emacs 26), so
> it's reasonable, IMO, and probably means the CC Mode contribution is
> relatively minor.

> The tricky part is testing the bidirectional part and the CC Mode part
> separately, by loading the old Emacs-23 CC Mode files into later
> versions of Emacs.  Maybe Alan could help by providing a Lisp file
> which, when loaded, would load CC Mode from a given directory -- this
> should be run before visiting any C/C++ file.

Here you go:

(defvar loaded-cc-version nil)
(defun load-cc-version (dir)
  "Load the version of CC Mode contained in directory DIR."
  (interactive "D")
  (or (string-match "/$" dir)
      (setq dir (concat dir "/")))
  (let (fname)
    ;; `c-fallback-style' gets stuff pushed onto it by loading cc-vars.  So
    ;; clear it out first, after a quick and dirty sanity check.
    (if (file-exists-p (concat dir "cc-vars.elc"))
        (setq c-fallback-style nil))
    (mapc (lambda (f)
            (condition-case err
                (load-file (setq fname (concat dir "cc-" f ".elc")))
              (error (message "Couldn't load %s" fname))))
          '("defs" "vars" "fix" "engine" "cmds" "menus"
            "align" "styles" "awk" "fonts" "mode"
            "subword" "bytecomp" "compat" "guess"))
    (setq loaded-cc-version dir)))

You can actually run load-cc-version repeatedly within an Emacs session
(I frequently do), together with M-x c-mode (etc.) on the pertinent
buffers.  Very occasionally, there're problems with defvars when their
value have changed between versions, since they don't get reinitialised.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Eli.

On Sat, Mar 28, 2020 at 12:08:30 +0300, Eli Zaretskii wrote:
> > Date: Sat, 28 Mar 2020 10:09:59 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org

> > > It would be interesting to compare 24 and 23, to see what slowdown
> > > there is.

> > I expect an average twofold slowdown due to bidirectional support.

> Actually, I take that back: in benchmarks such as scroll-up-benchmark,
> and in a buffer with predominantly ASCII text, the effect of
> bidirectional support should be very small, perhaps 10% or 20%.

> So the issue of testing old v23.x CC Mode with newer versions of Emacs
> becomes more important than I originally thought.  Just as a teaser,
> here's the profile I obtained by scrolling in Emacs 26.3 through the
> entire xdisp.c, it seems to point to c-find-decl-spots and its
> immediate callers and subroutines as the hot spots:

Yes.  They are the functions which search for and analyse declarations.
That they take so much of the processing time is more a reflection on
the complexity of the task rather than an opportunity for optimisation.
(Which isn't to say that further optimisation is impossible, just that
it won't be easy.)

> - command-execute                                                1446 100%
>  - call-interactively                                            1446 100%
>   - funcall-interactively                                        1446 100%
>    - execute-extended-command                                    1446 100%
>     - command-execute                                            1446 100%
>      - call-interactively                                        1446 100%
>       - funcall-interactively                                    1446 100%
>        - scroll-up-benchmark                                     1446 100%
>         - let                                                    1446 100%
>          - condition-case                                        1446 100%
>           - while                                                1446 100%
>            - redisplay                                           1351  93%
>             - redisplay_internal (C function)                    1071  74%
>              - jit-lock-function                                 1064  73%
>               - jit-lock-fontify-now                             1063  73%
>                - jit-lock--run-functions                         1063  73%
>                 - run-hook-wrapped                               1063  73%
>                  - #<compiled 0x2000000008356520>                1063  73%
>                   - font-lock-fontify-region                     1061  73%
>                    - c-font-lock-fontify-region                  1061  73%
>                     - font-lock-default-fontify-region            982  67%
>                      - font-lock-fontify-keywords-region          969  67%
>                       - c-font-lock-declarations                  686  47%
>                        - c-find-decl-spots                        674  46%
>                         - #<compiled 0x20000000077e07c0>          477  32%
>                          - c-get-fontification-context            253  17%
>                           - c-looking-at-or-maybe-in-bracelist    203  14%
>                            - c-backward-token-2                   196  13%
>                             - c-backward-sws                       35   2%
>                              - c-beginning-of-macro                12   0%
>                                 back-to-indentation                 6   0%
>                           - c-backward-token-2                     16   1%

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Date: Sat, 28 Mar 2020 11:43:01 +0000
> Cc: rms@gnu.org, rrandresf@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> (defvar loaded-cc-version nil)
> (defun load-cc-version (dir)
>   "Load the version of CC Mode contained in directory DIR."
>   (interactive "D")
>   (or (string-match "/$" dir)
>       (setq dir (concat dir "/")))
>   (let (fname)
>     ;; `c-fallback-style' gets stuff pushed onto it by loading cc-vars.  So
>     ;; clear it out first, after a quick and dirty sanity check.
>     (if (file-exists-p (concat dir "cc-vars.elc"))
>         (setq c-fallback-style nil))
>     (mapc (lambda (f)
>             (condition-case err
>                 (load-file (setq fname (concat dir "cc-" f ".elc")))
>               (error (message "Couldn't load %s" fname))))
>           '("defs" "vars" "fix" "engine" "cmds" "menus"
>             "align" "styles" "awk" "fonts" "mode"
>             "subword" "bytecomp" "compat" "guess"))
>     (setq loaded-cc-version dir)))

Thanks, but when I use this command to load CC Mode from Emacs 23.4,
and then visit xdisp.c, I get an error message:

  File mode specification error: (void-function make-local-hook)

and CC mode is not turned on.  What am I missing?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Eli.

On Sat, Mar 28, 2020 at 15:05:21 +0300, Eli Zaretskii wrote:
> > Date: Sat, 28 Mar 2020 11:43:01 +0000
> > Cc: rms@gnu.org, rrandresf@gmail.com, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > (defvar loaded-cc-version nil)
> > (defun load-cc-version (dir)
> >   "Load the version of CC Mode contained in directory DIR."
> >   (interactive "D")
> >   (or (string-match "/$" dir)
> >       (setq dir (concat dir "/")))
> >   (let (fname)
> >     ;; `c-fallback-style' gets stuff pushed onto it by loading cc-vars.  So
> >     ;; clear it out first, after a quick and dirty sanity check.
> >     (if (file-exists-p (concat dir "cc-vars.elc"))
> >         (setq c-fallback-style nil))
> >     (mapc (lambda (f)
> >             (condition-case err
> >                 (load-file (setq fname (concat dir "cc-" f ".elc")))
> >               (error (message "Couldn't load %s" fname))))
> >           '("defs" "vars" "fix" "engine" "cmds" "menus"
> >             "align" "styles" "awk" "fonts" "mode"
> >             "subword" "bytecomp" "compat" "guess"))
> >     (setq loaded-cc-version dir)))

> Thanks, but when I use this command to load CC Mode from Emacs 23.4,
> and then visit xdisp.c, I get an error message:

>   File mode specification error: (void-function make-local-hook)

> and CC mode is not turned on.  What am I missing?

make-local-hook was removed from Emacs in around 2010.  I think it still
exists in XEmacs (as much as anything can still be said to exist there).

If you can give me some time to eat my lunch, I'll seek out the
workaround I must have introduced some time after Emacs 23.4.  There may
be one or two other obsolete variables/functions which may now need
workarounds to allow timing of the old CC Mode versions.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Date: Sat, 28 Mar 2020 12:18:06 +0000
> Cc: rms@gnu.org, rrandresf@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> If you can give me some time to eat my lunch

Sure, not much point in programming on empty stomach ;-)

Thanks.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>   File mode specification error: (void-function make-local-hook)
>
> and CC mode is not turned on.  What am I missing?

(defalias 'make-local-hook #'ignore)


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>   18.59        n/a    0.320
>   23.4       5.436    1.019
>   24.5      25.482    1.636
>   25.3      22.370    1.610
>   26.3      16.533    1.911
>   27.0.90   17.602    2.339
>
> First column is the version, second and third column are the times with
> font-lock-mode enabled and disabled, respectively.

Thanks.  Some more questions need pondering before we can know what
might need to be done:
- What is the target?  is 5s good enough?  What about 10s?
- The middle column is basically a test of the font-lock performance of
  CC-mode's `c-mode`.  Is `c-mode` really all we care about here?
  [ I believe for the OP's case `c-mode` was just taken as an example
    but is not the only case he cares about, maybe not even the main one.  ]


-- Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Eli.

On Sat, Mar 28, 2020 at 15:28:50 +0300, Eli Zaretskii wrote:
> > Date: Sat, 28 Mar 2020 12:18:06 +0000
> > Cc: rms@gnu.org, rrandresf@gmail.com, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > If you can give me some time to eat my lunch

> Sure, not much point in programming on empty stomach ;-)

> Thanks.

Like Stefan said, aliasing make-local-hook to `ignore' will be fine.
All the relevant calls to add-hook set the LOCAL argument to t anyway.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Date: Sat, 28 Mar 2020 13:14:42 +0000
> Cc: rms@gnu.org, rrandresf@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> Like Stefan said, aliasing make-local-hook to `ignore' will be fine.

OK, using Emacs 23 CC Mode with Emacs 26.3 makes the scroll benchmark
run only 10% slower than in Emacs 23.  So it seems like most of the
slow-down in the scroll benchmark is due to font-lock of CC Mode.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> Like Stefan said, aliasing make-local-hook to `ignore' will be fine.
> All the relevant calls to add-hook set the LOCAL argument to t anyway.

... and even if they don't set the LOCAL argument, `make-local-hook`
still doesn't make any difference.

IOW, contrary to `setq` which sometimes sets the var buffer-locally and
sometimes does it globally depending on whether the var has been made
buffer-local, `add-hook` only uses the LOCAL argument to decide whether
the hook should be affected locally or globally (with the long-standing
ugly exception of those hooks that have been `make-local-variable`,
which I'd love to get rid of).  I changed this back in Emacs-21 ;-)


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Clément Pit-Claudel]

On 28/03/2020 05.58, Eli Zaretskii wrote:
> I think the way to increase performance of C/C++ fontification,
> indentation, etc. is to use syntactic analysis in C (or another
> similarly fast compiled language).  Several programs/libraries are
> available out there for this purpose, and we should encourage
> interested individuals to work on making these used by our major modes
> that support programming languages.  The time we could do that with
> ad-hoc regexps and the likes of syntax-ppss has gone, IMO.

I've looked, but I don't really know of any such libraries.  The problem that cc-mode has to solve most of the time is fontifying an invalid C program (since, unless the programmer is using a structural editing mode), code that is being typed is invalid most of the time.  Most language parsing facilities out there do not have great error recovery, so they are not very good at giving semantic annotations useful for syntax highlighting broken code.

The only great example I know is the OCaml parser written in Menhir, which is incredibly robust.  The most extreme one I know in the other direction is agda-mode, which only displays syntax highlighting when the buffer contains syntactically valid, type-correct code.

I wonder if Stefan's smie could be used to do syntax highlighting.

Clément.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Sat, 28 Mar 2020 10:43:00 -0400
> 
> On 28/03/2020 05.58, Eli Zaretskii wrote:
> > I think the way to increase performance of C/C++ fontification,
> > indentation, etc. is to use syntactic analysis in C (or another
> > similarly fast compiled language).  Several programs/libraries are
> > available out there for this purpose, and we should encourage
> > interested individuals to work on making these used by our major modes
> > that support programming languages.  The time we could do that with
> > ad-hoc regexps and the likes of syntax-ppss has gone, IMO.
> 
> I've looked, but I don't really know of any such libraries.

What about tree-sitter?

> The problem that cc-mode has to solve most of the time is fontifying an invalid C program (since, unless the programmer is using a structural editing mode), code that is being typed is invalid most of the time.  Most language parsing facilities out there do not have great error recovery, so they are not very good at giving semantic annotations useful for syntax highlighting broken code.

AFAIK, tree-sitter's use cases include the above.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Clément Pit-Claudel]

On 28/03/2020 11.03, Eli Zaretskii wrote:
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Sat, 28 Mar 2020 10:43:00 -0400
>>
>> On 28/03/2020 05.58, Eli Zaretskii wrote:
>>> I think the way to increase performance of C/C++ fontification,
>>> indentation, etc. is to use syntactic analysis in C (or another
>>> similarly fast compiled language).  Several programs/libraries are
>>> available out there for this purpose, and we should encourage
>>> interested individuals to work on making these used by our major modes
>>> that support programming languages.  The time we could do that with
>>> ad-hoc regexps and the likes of syntax-ppss has gone, IMO.
>>
>> I've looked, but I don't really know of any such libraries.
> 
> What about tree-sitter?

Neat! It looks promising.  I forgot about this one.
It seems that Atom uses it, so that's a pretty strong endorsement too!
Could it be used with Emacs 25's modules? Otherwise, how can users define new languages without recompiling Emacs?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Sat, 28 Mar 2020 11:17:34 -0400
> 
> > What about tree-sitter?
> 
> Neat! It looks promising.  I forgot about this one.
> It seems that Atom uses it, so that's a pretty strong endorsement too!
> Could it be used with Emacs 25's modules? Otherwise, how can users define new languages without recompiling Emacs?

I'd start building Emacs with that library, and figuring out how to
interface it from Lisp for our purposes.  Only after that I'd worry
about making that a module and/or the ability to add languages.  "The
perfect is the enemy of the good", and many languages already have
support files ready to be used.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 28.03.2020 17:17, Clément Pit-Claudel wrote:
> Could it be used with Emacs 25's modules?

Apparently so: https://github.com/karlotness/tree-sitter.el

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 28.03.2020 18:10, Dmitry Gutov wrote:
> On 28.03.2020 17:17, Clément Pit-Claudel wrote:
>> Could it be used with Emacs 25's modules?
> 
> Apparently so: https://github.com/karlotness/tree-sitter.el

Sorry, this one is more active: 
https://github.com/ubolonton/emacs-tree-sitter

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 28 Mar 2020 18:11:05 +0200
> Cc: emacs-devel@gnu.org
> 
> On 28.03.2020 18:10, Dmitry Gutov wrote:
> > On 28.03.2020 17:17, Clément Pit-Claudel wrote:
> >> Could it be used with Emacs 25's modules?
> > 
> > Apparently so: https://github.com/karlotness/tree-sitter.el
> 
> Sorry, this one is more active: 
> https://github.com/ubolonton/emacs-tree-sitter

Do any of those support font-lock?  Because I think that's the context
of Clément's question about modules.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 28.03.2020 18:25, Eli Zaretskii wrote:
>> From: Dmitry Gutov<dgutov@yandex.ru>
>> Date: Sat, 28 Mar 2020 18:11:05 +0200
>> Cc:emacs-devel@gnu.org
>>
>> On 28.03.2020 18:10, Dmitry Gutov wrote:
>>> On 28.03.2020 17:17, Clément Pit-Claudel wrote:
>>>> Could it be used with Emacs 25's modules?
>>> Apparently so:https://github.com/karlotness/tree-sitter.el
>> Sorry, this one is more active:
>> https://github.com/ubolonton/emacs-tree-sitter
> Do any of those support font-lock?  Because I think that's the context
> of Clément's question about modules.

Apparently they aim to replace it with a new specialized jit-highlighter:

https://github.com/ubolonton/emacs-tree-sitter/issues/4
https://github.com/ubolonton/emacs-tree-sitter/pull/16

But that's not due to modules' limitations, but because of font-lock's 
limitations (allegedly).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[andrés ramírez]

Hi Richard.

Richard> To tell the people with the strongest commitment to freedom
Richard> to use an old version of Emacs is to choose failure.  We need
Richard> to re-optimize the Emacs master so it gives good performance
Richard> on those machines.

We Agree on that.

>> BTW. On my comparissons that variable was set no nil as Doc says:
>> --8<---------------cut here---------------start------------->8---
>> If nil, use the default decoration (typically the minimum
>> available).  --8<---------------cut
>> here---------------end--------------->8--- Also xdisp.c. Just load
>> c-mode not c++-mode

Richard> What were the results of the comparisons you did with this
Richard> setting?  (I wasn't following this topic until yesterday.)

It was on the first email of this thread:
--8<---------------cut here---------------start------------->8---
https://lists.gnu.org/archive/html/emacs-devel/2020-03/msg00511.html
--8<---------------cut here---------------end--------------->8---

Ulrich Mueller has a new comparisson with an up to date machine. On his
comparisson he has it with font-lock enable and disable. I am going to
do the same an update the table accordingly.
--8<---------------cut here---------------start------------->8---
https://lists.gnu.org/archive/html/emacs-devel/2020-03/msg00724.html
--8<---------------cut here---------------end--------------->8---

>> With the help of Michael Albinus I have update the emacs23 builtin
>> tramp version to the latest one that supports GNU/Emacs23

Richard> Has Tramp become a lot slower?  Or is it only that the
Richard> current Tramp doesn't work on Emacs 23?

Michael Albinus has answered on this email:
--8<---------------cut here---------------start------------->8---
https://lists.gnu.org/archive/html/emacs-devel/2020-03/msg00721.html
--8<---------------cut here---------------end--------------->8---

Richard> Has Org mode become a lot slower?  Or is it only that the
Richard> current Org mode doesn't work on Emacs 23?

Current org-mode does not work on Emacs23. But I remember You were part
of that discussion in the past. And the conclusion of that discussion
was to support just three versions. But the org-mode guys replied a
shortage of man power for it.

Best Regards

Re: emacs rendering comparisson between emacs23 and emacs26.3

[rrandresf]

Hi Guys.

Dmitry> Apparently they aim to replace it with a new specialized
Dmitry> jit-highlighter:

I was just remembering the AA-tree on the noverlays branch(from Andreas Politz). I do not
remember the sender but a guy was trying to save and merge code from
that branch to master (before closing the Emacs27 merge window). I think
the idea with this AA-tree was improving actual font-lock with the
AA-tree. I think a jit-highlighter would render with more
performance. Am I right?.

Best Regards
ps: the sender was not successfull on using an AA-tree. But Andreas was.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 28.03.2020 23:08, rrandresf@gmail.com wrote:
> I think
> the idea with this AA-tree was improving actual font-lock with the
> AA-tree.

I don't think so. IIUC, the branch's goal was to improve the performance 
of overlays. Not text properties (which font-lock uses).

> I think a jit-highlighter would render with more
> performance.

The current font-lock engine is jit-lock, and it's usually not the 
bottleneck of our syntax functionality. The font-lock rules themselves 
are generally more CPU-intensive.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[andrés ramírez]

Hi Dmitry.

Dmitry> I don't think so. IIUC, the branch's goal was to improve the
Dmitry> performance of overlays. Not text properties (which font-lock
Dmitry> uses).

What about font-lock changing its use of text properties to use in place
overlays?

Best Regards

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 28.03.2020 23:38, andrés ramírez wrote:
> Dmitry> I don't think so. IIUC, the branch's goal was to improve the
> Dmitry> performance of overlays. Not text properties (which font-lock
> Dmitry> uses).
> 
> What about font-lock changing its use of text properties to use in place
> overlays?

Not sure I understand the question. Could you rephrase that?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[andres.ramirez]

Hi Dmitry.

Dmitry> don't think
Dmitry> so. IIUC, the branch's goal was to improve the performance of
Dmitry> overlays. Not text properties (which font-lock uses).
>> 
>> What about font-lock changing its use of text properties to use in
>> place overlays?

Dmitry> Not sure I understand the question. Could you rephrase that?

Now: font-lock uses text properties
Posibility: font-lock uses overlays (dismiss text properties)

the AA-tree could be usable on that Posibility?

Best Regards

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 29.03.2020 00:15, andres.ramirez wrote:
> Now: font-lock uses text properties
> Posibility: font-lock uses overlays (dismiss text properties)
> 
> the AA-tree could be usable on that Posibility?

Yes. But then, I'm not sure why you'd want that change.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > 23.4       5.436    1.019
    > 24.5      25.482    1.636
    > 25.3      22.370    1.610
    > 26.3      16.533    1.911
    > 27.0.90   17.602    2.339

  > First column is the version, second and third column are the times with
  > font-lock-mode enabled and disabled, respectively.

What this says is that even with font lock disabled, there has been a
lot of slowdown.  About half of the slowdown was in version 24, and
most of the rest was after version 26.

If someone can figure out why, we may be able to speed it up again.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[andrés ramírez]

Hi Guys.

Eli> OK, using Emacs 23 CC Mode with Emacs 26.3 makes the scroll
Eli> benchmark run only 10% slower than in Emacs 23.  So it seems like
Eli> most of the slow-down in the scroll benchmark is due to font-lock
Eli> of CC Mode.

This my updated benchmark:

Year 2016 (allwinner H3 Quad Core CPU @ 1GHz). In all cases, I
used xdisp.c from version 26.3 as the test file.

| EMACS      |       no font-lock | cc-mode from 23.4 (without cc-guess.el) |
|------------+--------------------+-----------------------------------------|
| emacs-23.4 | 22.041109085083008 |                       131.7112021446228 |
| emacs-26-3 |  34.09645438194275 |                       111.6537675857544 |

| EMACS 26.3 |      no font-lock |  built-in cc-mode |
|------------+-------------------+-------------------|
|  | 35.47433280944824 | 261.2694363594055 |

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: cpitclaudel@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 28 Mar 2020 18:43:13 +0200
> 
> >>> Apparently so:https://github.com/karlotness/tree-sitter.el
> >> Sorry, this one is more active:
> >> https://github.com/ubolonton/emacs-tree-sitter
> > Do any of those support font-lock?  Because I think that's the context
> > of Clément's question about modules.
> 
> Apparently they aim to replace it with a new specialized jit-highlighter:
> 
> https://github.com/ubolonton/emacs-tree-sitter/issues/4
> https://github.com/ubolonton/emacs-tree-sitter/pull/16
> 
> But that's not due to modules' limitations, but because of font-lock's 
> limitations (allegedly).

I'm not sure what limitations they have in mind: font-lock supports
functions, not just regular expressions.  Why does it have to be
replaced?

In any case, tossing JIT font-lock and going back to using all kinds
of hooks is a non-starter.  Emacs 19 and 20 did that, and we found
this method to have fundamental problems which couldn't be resolved in
satisfactory ways.  That is why Emacs 21 introduced jit-lock.el and
the associated support in the display engine for fontifying what is
about to be displayed.  Going back to hooks (if that is what they
intend to do; I couldn't be sure from reading the discussion) is a
step in the wrong direction.

Likewise using overlays instead of text properties (again, not sure
they actually intend doing that, maybe they just talked about it) is
wrong even if we assume the tree-based implementation that sits on a
branch.

More generally, I'm surprised, to say the least, that such a
significant job is being done without saying even a single word here,
let alone making sure the basic design is deemed reasonable by the
core developers.  Do they really think this will make the acceptance
of the code easier?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> In any case, tossing JIT font-lock and going back to using all kinds
> of hooks is a non-starter.

I think it's just a reflection of the fact that author doesn't know
enough about jit-lock and font-lock.

I sent him some patches that should help.

> More generally, I'm surprised, to say the least, that such
> a significant job is being done without saying even a single word
> here, let alone making sure the basic design is deemed reasonable by
> the core developers.  Do they really think this will make the
> acceptance of the code easier?

I think this risks being read as unwelcoming, which is unlikely to
encourage them to get in touch with emacs-devel.

tree-sitter, like LSP, is something Emacs should embrace.  I think in
the case of tree-sitter we would benefit from participating directly in
its development to make sure it's easy enough to integrate into Emacs
(e.g. doesn't require a Javascript or C compiler on the end-user side).


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 29 Mar 2020 12:39:55 -0400
> Cc: cpitclaudel@gmail.com, emacs-devel@gnu.org, Dmitry Gutov <dgutov@yandex.ru>
> tree-sitter, like LSP, is something Emacs should embrace.

  https://lists.gnu.org/archive/html/emacs-devel/2020-01/msg00059.html

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 29.03.2020 19:39, Stefan Monnier wrote:
> I think it's just a reflection of the fact that author doesn't know
> enough about jit-lock and font-lock.

Indeed, that font-lock rule can be a function is not very well-known.

> I sent him some patches that should help.

That's mighty good of you.

I was curious and wanted to take a look, though, but couldn't find them 
neither among pull requests, nor the issue comments. :-(

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 29.03.2020 17:30, Eli Zaretskii wrote:

>> Apparently they aim to replace it with a new specialized jit-highlighter:
>>
>> https://github.com/ubolonton/emacs-tree-sitter/issues/4
>> https://github.com/ubolonton/emacs-tree-sitter/pull/16
>>
>> But that's not due to modules' limitations, but because of font-lock's
>> limitations (allegedly).
> 
> I'm not sure what limitations they have in mind: font-lock supports
> functions, not just regular expressions.  Why does it have to be
> replaced?

Hence "allegedly". I haven't looked into this issue, or this project, 
deep enough to say more.

> In any case, tossing JIT font-lock and going back to using all kinds
> of hooks is a non-starter.  Emacs 19 and 20 did that, and we found
> this method to have fundamental problems which couldn't be resolved in
> satisfactory ways.  That is why Emacs 21 introduced jit-lock.el and
> the associated support in the display engine for fontifying what is
> about to be displayed.  Going back to hooks (if that is what they
> intend to do; I couldn't be sure from reading the discussion)

I think they're still experimenting. But whatever they come up with in 
that alternative jit highlighter, could hopefully be ported to jit-lock.

> Likewise using overlays instead of text properties (again, not sure
> they actually intend doing that, maybe they just talked about it) is
> wrong even if we assume the tree-based implementation that sits on a
> branch.

Yup.

> More generally, I'm surprised, to say the least, that such a
> significant job is being done without saying even a single word here,
> let alone making sure the basic design is deemed reasonable by the
> core developers.

There is no point in taking offense at that. I think this is just 
another case of the extended developer community, and especially 
newcomers, being uncomfortable with our old-timey tools. I'm pretty sure 
none of the participants in that project even read your earlier posting 
to emacs-devel about tree-sitter.

And if you ask some of them why they didn't think to ask on the mailing 
list first, the answers will likely be very benign like the story here: 
https://www.reddit.com/r/emacs/comments/fqji2t/any_elisp_projects_looking_for_contributors/flqs6v9/

> Do they really think this will make the acceptance
> of the code easier?

They really hope they won't have to ask.

It's a blessing and a curse of Emacs's extensibility. If this project is 
successful, it is likely to reside entirely outside of Emacs core. Just 
like the lsp-mode project lives outside, and only filed a couple of bug 
reports when the built-in JSON support (which they really have to use) 
showed insufficient performance characteristics.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: cpitclaudel@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 29 Mar 2020 21:27:50 +0300
> 
> If this project is successful, it is likely to reside entirely
> outside of Emacs core. Just like the lsp-mode project lives outside,
> and only filed a couple of bug reports when the built-in JSON
> support (which they really have to use) showed insufficient
> performance characteristics.

FWIW, I don't think it's reasonable to have such core functionality
outside of core.  Such external projects can never last as long as
Emacs did and will, because there's not enough resources to sustain
them.  Bringing them into core is the only way to both make them clean
enough and tested enough, and make sure they stay alive for many
years, let alone make sure applications are built based on them.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[조성빈]

> 2020. 3. 29. 오후 11:32, Eli Zaretskii <eliz@gnu.org> 작성:
> 
> 
>> 
>> Cc: cpitclaudel@gmail.com, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Sat, 28 Mar 2020 18:43:13 +0200
>> 
>>>>> Apparently so:https://github.com/karlotness/tree-sitter.el
>>>> Sorry, this one is more active:
>>>> https://github.com/ubolonton/emacs-tree-sitter
>>> Do any of those support font-lock?  Because I think that's the context
>>> of Clément's question about modules.
>> 
>> Apparently they aim to replace it with a new specialized jit-highlighter:
>> 
>> https://github.com/ubolonton/emacs-tree-sitter/issues/4
>> https://github.com/ubolonton/emacs-tree-sitter/pull/16
>> 
>> But that's not due to modules' limitations, but because of font-lock's 
>> limitations (allegedly).
> 
> I'm not sure what limitations they have in mind: font-lock supports
> functions, not just regular expressions.  Why does it have to be
> replaced?

According to them, font-lock doesn’t provide enough control. This is the exact text quoted from the linked issue:

>> so, we don't need to reinvent the wheel but neither rely on font-lock in a way that limit us, as I see it, we can implement our own syntax highlight functions for each grammar available and let font-lock use them instead of the defaults. Unless, of course, I'm missing something.
> 
> That only covers how to highlight a region. I think the main complexity is in the change-handling mechanisms, i.e. which regions should be re-highlighted, when.

I don’t know about font-lock enough to say whether this is true or not. 

> In any case, tossing JIT font-lock and going back to using all kinds
> of hooks is a non-starter.  Emacs 19 and 20 did that, and we found
> this method to have fundamental problems which couldn't be resolved in
> satisfactory ways.  That is why Emacs 21 introduced jit-lock.el and
> the associated support in the display engine for fontifying what is
> about to be displayed.  Going back to hooks (if that is what they
> intend to do; I couldn't be sure from reading the discussion) is a
> step in the wrong direction.
> 
> Likewise using overlays instead of text properties (again, not sure
> they actually intend doing that, maybe they just talked about it) is
> wrong even if we assume the tree-based implementation that sits on a
> branch.
> 
> More generally, I'm surprised, to say the least, that such a
> significant job is being done without saying even a single word here,
> let alone making sure the basic design is deemed reasonable by the
> core developers.  Do they really think this will make the acceptance
> of the code easier?

Actually I’m surprised that people here on the mailing list expects people to ask here — I think most people aren’t even aware of this list, and even if they do, they prefer working on Github + MELPA, with the community. 
I’m pretty sure they aren’t aiming to merge the code into Emacs — they’re in to write another external package, like lsp-mode or magit.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 29.03.2020 21:39, Eli Zaretskii wrote:
>> Cc: cpitclaudel@gmail.com, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Sun, 29 Mar 2020 21:27:50 +0300
>>
>> If this project is successful, it is likely to reside entirely
>> outside of Emacs core. Just like the lsp-mode project lives outside,
>> and only filed a couple of bug reports when the built-in JSON
>> support (which they really have to use) showed insufficient
>> performance characteristics.
> 
> FWIW, I don't think it's reasonable to have such core functionality
> outside of core.  Such external projects can never last as long as
> Emacs did and will, because there's not enough resources to sustain
> them.  Bringing them into core is the only way to both make them clean
> enough and tested enough, and make sure they stay alive for many
> years, let alone make sure applications are built based on them.

I'd also rather it benefits all our users, and the development can take 
advantage of our experience.

But this is the scenario I've seen many times. And yes, with several 
innovative projects abandoned after a few years.

Collaborating with the core developers seems objectively more difficult 
for many such projects, however.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: 조성빈 <pcr910303@icloud.com>
> Date: Mon, 30 Mar 2020 04:04:08 +0900
> Cc: Dmitry Gutov <dgutov@yandex.ru>, cpitclaudel@gmail.com,
>  Emacs-devel@gnu.org
> 
> > I'm not sure what limitations they have in mind: font-lock supports
> > functions, not just regular expressions.  Why does it have to be
> > replaced?
> 
> According to them, font-lock doesn’t provide enough control. This is the exact text quoted from the linked issue:
> 
> >> so, we don't need to reinvent the wheel but neither rely on font-lock in a way that limit us, as I see it, we can implement our own syntax highlight functions for each grammar available and let font-lock use them instead of the defaults. Unless, of course, I'm missing something.
> > 
> > That only covers how to highlight a region. I think the main complexity is in the change-handling mechanisms, i.e. which regions should be re-highlighted, when.

But that's exactly the problem that jit-lock solves, and solves well.
Because only the display engine knows exactly what parts need to be
re-highlighted and when.  IOW, this problem is already solved, all
that's needed is to play by jit-lock's rules.

> > More generally, I'm surprised, to say the least, that such a
> > significant job is being done without saying even a single word here,
> > let alone making sure the basic design is deemed reasonable by the
> > core developers.  Do they really think this will make the acceptance
> > of the code easier?
> 
> Actually I’m surprised that people here on the mailing list expects people to ask here — I think most people aren’t even aware of this list, and even if they do, they prefer working on Github + MELPA, with the community. 

I'm quite sure that everyone knows where the Emacs maintainers can be
contacted.

> I’m pretty sure they aren’t aiming to merge the code into Emacs — they’re in to write another external package, like lsp-mode or magit.

As I wrote in another message, I don't believe doing so is viable for
a core feature such as font-lock or major-mode support in general.
Magit is different, since it's an application, not a core feature
designed to be basis for applications.  And if LSP mode wants a long
life and many applications built upon it, it should come to Emacs core
as well, at least IMNSHO.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > To tell the people with the strongest commitment to freedom to use an
  > > old version of Emacs is to choose failure.  We need to re-optimize the
  > > Emacs master so it gives good performance on those machines.

  > That is of course the ideal we should always strive to, but it's
  > unreasonable to expect newer versions of Emacs to stay as fast as the
  > oder ones: we do add functionality, and some of that eats up CPU
  > cycles.

Would you please strive to keep Emacs from slowing down
for users who don't use, or who disable, the new functionality?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org
> Date: Sun, 29 Mar 2020 23:42:05 -0400
> 
>   > That is of course the ideal we should always strive to, but it's
>   > unreasonable to expect newer versions of Emacs to stay as fast as the
>   > oder ones: we do add functionality, and some of that eats up CPU
>   > cycles.
> 
> Would you please strive to keep Emacs from slowing down
> for users who don't use, or who disable, the new functionality?

For new features that can be disabled, that is mostly the case
anyway.  But most new features we add don't have such knobs, usually
because we don't envision anyone to want them.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Would you please strive to keep Emacs from slowing down
  > > for users who don't use, or who disable, the new functionality?

  > For new features that can be disabled, that is mostly the case
  > anyway.  But most new features we add don't have such knobs, usually
  > because we don't envision anyone to want them.

My point is, if a feature causes significant slowdown, in general usage,
that in itself is a reason to give it a knob.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org
> Date: Mon, 30 Mar 2020 22:28:10 -0400
> 
>   > For new features that can be disabled, that is mostly the case
>   > anyway.  But most new features we add don't have such knobs, usually
>   > because we don't envision anyone to want them.
> 
> My point is, if a feature causes significant slowdown, in general usage,
> that in itself is a reason to give it a knob.

When we know a new feature causes significant slowdown, we either fix
it, or provide a way to disable or work around it.  The problem is, we
don't always know there's slowdown, as it frequently happens only in
specific rare use cases.  (John said some time ago we should have a
benchmarking test suite, but I don't think anyone's working on it.)
Also, several slowdowns each of which is minor can together cause a
significant slowdown, and in that case it is even harder to detect and
fix that.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

> When we know a new feature causes significant slowdown, we either fix
> it, or provide a way to disable or work around it.  The problem is, we
> don't always know there's slowdown, as it frequently happens only in
> specific rare use cases.  (John said some time ago we should have a
> benchmarking test suite, but I don't think anyone's working on it.)

https://elpa.gnu.org/packages/elisp-benchmarks.html ?

  Andrea

-- 
akrl@sdf.org

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: rms@gnu.org, acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org
> Date: Tue, 31 Mar 2020 16:05:08 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > When we know a new feature causes significant slowdown, we either fix
> > it, or provide a way to disable or work around it.  The problem is, we
> > don't always know there's slowdown, as it frequently happens only in
> > specific rare use cases.  (John said some time ago we should have a
> > benchmarking test suite, but I don't think anyone's working on it.)
> 
> https://elpa.gnu.org/packages/elisp-benchmarks.html ?

(Any idea why this isn't in the ELPA repository?)

This might be a beginning, but the issue at hand is not to benchmark
Emacs Lisp programs, it is to benchmark Emacs commands, so we could
make sure we don't have speed regressions.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: rms@gnu.org, acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org
>> Date: Tue, 31 Mar 2020 16:05:08 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > When we know a new feature causes significant slowdown, we either fix
>> > it, or provide a way to disable or work around it.  The problem is, we
>> > don't always know there's slowdown, as it frequently happens only in
>> > specific rare use cases.  (John said some time ago we should have a
>> > benchmarking test suite, but I don't think anyone's working on it.)
>> 
>> https://elpa.gnu.org/packages/elisp-benchmarks.html ?
>
> (Any idea why this isn't in the ELPA repository?)

What do you mean?  I think it is (as external package tho).

> This might be a beginning, but the issue at hand is not to benchmark
> Emacs Lisp programs, it is to benchmark Emacs commands, so we could
> make sure we don't have speed regressions.

Yes is just a start.  I think everything is in Emacs core can be
classified as elisp implementation, I would not know where to draw a
line otherwise.

We could add benchmark categories if we start having a number of and we
want to better separate them.

  Andrea

-- 
akrl@sdf.org

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: rms@gnu.org, acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org
> Date: Tue, 31 Mar 2020 18:00:25 +0000
> 
> >> https://elpa.gnu.org/packages/elisp-benchmarks.html ?
> >
> > (Any idea why this isn't in the ELPA repository?)
> 
> What do you mean?  I think it is (as external package tho).

When I checkout the ELPA repository, I don't seem to have it.  Or did
I miss something?

> > This might be a beginning, but the issue at hand is not to benchmark
> > Emacs Lisp programs, it is to benchmark Emacs commands, so we could
> > make sure we don't have speed regressions.
> 
> Yes is just a start.  I think everything is in Emacs core can be
> classified as elisp implementation, I would not know where to draw a
> line otherwise.

I said "commands", and it wasn't an accident.  Those are user-visible
units, so IMO we should start by having benchmarks for frequent
commands.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: rms@gnu.org, acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org
>> Date: Tue, 31 Mar 2020 18:00:25 +0000
>> 
>> >> https://elpa.gnu.org/packages/elisp-benchmarks.html ?
>> >
>> > (Any idea why this isn't in the ELPA repository?)
>> 
>> What do you mean?  I think it is (as external package tho).
>
> When I checkout the ELPA repository, I don't seem to have it.  Or did
> I miss something?

Is in the 'externals/elisp-benchmarks' branch.

The ELPA readme explain this (line 231).

>> > This might be a beginning, but the issue at hand is not to benchmark
>> > Emacs Lisp programs, it is to benchmark Emacs commands, so we could
>> > make sure we don't have speed regressions.
>> 
>> Yes is just a start.  I think everything is in Emacs core can be
>> classified as elisp implementation, I would not know where to draw a
>> line otherwise.
>
> I said "commands", and it wasn't an accident.  Those are user-visible
> units, so IMO we should start by having benchmarks for frequent
> commands.

What I'm saying is that I agree with you and I think these new
benchmarks should go in the 'elisp-benchmarks' package too.  Perhaps we
can classify these but I think is good to have one suite collecting all
benchmarks.

Andrea

-- 
akrl@sdf.org

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>> What do you mean?  I think it is (as external package tho).
> When I checkout the ELPA repository, I don't seem to have it.  Or did
> I miss something?

You probably need to do

    make externals

> I said "commands", and it wasn't an accident.  Those are user-visible
> units, so IMO we should start by having benchmarks for frequent
> commands.

Indeed, none of the current benchmarks exercise the en/decoding code, or
the rendering code, or the process I/O code, ...



        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > When we know a new feature causes significant slowdown, we either fix
  > it, or provide a way to disable or work around it.  The problem is, we
  > don't always know there's slowdown, as it frequently happens only in
  > specific rare use cases.

1. If the slowdown cases are truly rare, I don't think they will bother
users very often.

2. I think we should be able to anticipate which changes _might_ cause
noticeable slowdowns.  They will be changes that will come into play
frequently in basic activities such as cursor motion, basic editing
operations, or redisplay.  We may not anticipate 100%, but let's start
trying.


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org
> Date: Tue, 31 Mar 2020 22:07:38 -0400
> 
>   > When we know a new feature causes significant slowdown, we either fix
>   > it, or provide a way to disable or work around it.  The problem is, we
>   > don't always know there's slowdown, as it frequently happens only in
>   > specific rare use cases.
> 
> 1. If the slowdown cases are truly rare, I don't think they will bother
> users very often.

A frequent case is that the use case is rare on the average, but for
some user(s) it is not so rare.

> 2. I think we should be able to anticipate which changes _might_ cause
> noticeable slowdowns.  They will be changes that will come into play
> frequently in basic activities such as cursor motion, basic editing
> operations, or redisplay.  We may not anticipate 100%, but let's start
> trying.

I think we do try: patch review frequently has these aspects
discussed.  But it is a well known fact that humans are lousy
predictors of code speed, which is why the advice is always to profile
before deciding where to optimize.  I think we miss the slowdowns for
the same reason.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > 1. If the slowdown cases are truly rare, I don't think they will bother
  > > users very often.

  > A frequent case is that the use case is rare on the average, but for
  > some user(s) it is not so rare.

Yes.  I don't think we can attain perfection, so I'm suggesting a guideline
so we can do better overall without too much extra work.

  > I think we do try: patch review frequently has these aspects
  > discussed.  But it is a well known fact that humans are lousy
  > predictors of code speed.

I'm not suggesting we try to _predict_ the effect on code speed,
rather than we have a guideline to identify the cases where _if_
the code gets slower it would be hard for a user to escape that.

What we would do, in those cases, is add a switch to turn off
the changes.



-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > What we would do, in those cases, is add a switch to turn off
 > the changes.

What we did, in those cases, was to remove the one major switch to turn
off those changes - 'open-paren-in-column-0-is-defun-start'.  With the
argument that people want to comment out larger regions of code by
simply putting comment delimiters at the beginning and end of that code.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org
> Date: Wed, 01 Apr 2020 22:32:12 -0400
> 
>   > I think we do try: patch review frequently has these aspects
>   > discussed.  But it is a well known fact that humans are lousy
>   > predictors of code speed.
> 
> I'm not suggesting we try to _predict_ the effect on code speed,
> rather than we have a guideline to identify the cases where _if_
> the code gets slower it would be hard for a user to escape that.
> 
> What we would do, in those cases, is add a switch to turn off
> the changes.

Without some efficient method of identifying the cases where this is
probable, I fear we will have to add such switches for almost every
non-trivial change (and for some trivial ones as well).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > What we did, in those cases, was to remove the one major switch to turn
  > off those changes - 'open-paren-in-column-0-is-defun-start'.

Can you pleese explain what you mean?

What exactly was "removed"?  The variable
'open-paren-in-column-0-is-defun-start' seems to still exist, and its
default value seems to be t (my .emacs does not set it and it is t).


								  With the
  > argument that people want to comment out larger regions of code by
  > simply putting comment delimiters at the beginning and end of that code.

You have not said concretely which practice this refers to, but the
feature of special meaning for open-paren in column zero affects Emacs
editing commands, not the meaning of the program when executed.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I'm not suggesting we try to _predict_ the effect on code speed,
  > > rather than we have a guideline to identify the cases where _if_
  > > the code gets slower it would be hard for a user to escape that.
  > > 
  > > What we would do, in those cases, is add a switch to turn off
  > > the changes.

  > Without some efficient method of identifying the cases where this is
  > probable, I fear we will have to add such switches for almost every
  > non-trivial change (and for some trivial ones as well).

I am not suggesting that we band over backwards.  But it would be useful
to err on the side of letting people turn these features off.

We don't have to preserve those switches forever.  After the feature
is included in a release for several months, we can ask the users,
"Who finds it useful to disable the XYZ feature?"  If nobody reports a
real need for that switch, we can delete it.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > Can you pleese explain what you mean?
 >
 > What exactly was "removed"?  The variable
 > 'open-paren-in-column-0-is-defun-start' seems to still exist, and its
 > default value seems to be t (my .emacs does not set it and it is t).
 >
 >
 > 								  With the
 >    > argument that people want to comment out larger regions of code by
 >    > simply putting comment delimiters at the beginning and end of that code.
 >
 > You have not said concretely which practice this refers to, but the
 > feature of special meaning for open-paren in column zero affects Emacs
 > editing commands, not the meaning of the program when executed.

'open-paren-in-column-0-is-defun-start', if non-nil, conceptually allows
progmodes to avoid scanning an entire buffer in order to get things like
syntax highlighting and code indenting right.  Rather, progmodes are
allowed to find the first or next paren in column zero wrt a given
position and base further decisions on the assumption that such a paren
is on the "top level" of its buffer.

Recent Emacsen either ignore that variable or silently reset it to nil
internally so it doesn't get into their way.  Their progmodes either
always scan an entire buffer from its beginning or use some elaborate,
fragile techniques to find such a top level position.  Moreover, our
underlying mechanism for syntax highlighting always marks the entire
rest of a buffer as dirty after every single editing change.  This has
the consequence that that entire part has to be continuously rescanned
when some of it is shown in another window.

The basic slowness of Emacs over the past years is a direct consequence
of that policy.  Most people don't care because they are using fast
processors that easily compensate the resulting overhead.  People with
old and slow hardware suffer from it.

So since you earlier asked for "a switch to turn off the changes" then
my answer is that such a switch already exists but has been deactivated.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> The basic slowness of Emacs over the past years is a direct consequence
> of that policy.

Do you have some kind of (concrete, statistical, circumstantial, ...)
evidence for that?


        Stefan "whose fastest machine has an Intel i3-4170"

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >> The basic slowness of Emacs over the past years is a direct consequence
 >> of that policy.
 >
 > Do you have some kind of (concrete, statistical, circumstantial, ...)
 > evidence for that?

I'd gladly provide such evidence to prove or disprove my claim.  But
customizing 'open-paren-in-column-0-is-defun-start' has no effect here
any more and reconstructing its earlier semantics would consume more
resources than I can afford.

 >          Stefan "whose fastest machine has an Intel i3-4170"

martin "on an AMD Athlon 64 X2 Dual Core Processor 5000+"

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>>> The basic slowness of Emacs over the past years is a direct consequence
>>> of that policy.
>> Do you have some kind of (concrete, statistical, circumstantial, ...)
>> evidence for that?
> I'd gladly provide such evidence to prove or disprove my claim.

I understand proving/disproving it is hard.

I'm really asking what are the signs that make you think it's
due to the lack of use of `open-paren-in-column-0-is-defun-start`.

> But customizing 'open-paren-in-column-0-is-defun-start' has no effect
> here any more and reconstructing its earlier semantics would consume
> more resources than I can afford.

Yes, I don't think there's much interested in going back to that.
OTOH if we could find specific cases where the old code was faster
(supposedly due to its use of `open-paren-in-column-0-is-defun-start`),
we could think about how to recover that performance.

E.g. `open-paren-in-column-0-is-defun-start` was used in
`forward-comment` was scanning comments backward to avoid having to use
`parse-partial-sexp` all the way from `point-min` in the worst case.
We replaced that with a parse cache (`syntax-ppss`) which seems to be
just as good in my experience (and significantly better in those
languages where open parens in column 0 are virtually non-existent).
If we can show that this replacement is not "just as good", there are
various options to improve it or replace it, but we'd need to know which
case(s) matter in order to design it.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

On Fri, Apr 03, 2020 at 08:59:54 +0200, martin rudalics wrote:
>  > Can you pleese explain what you mean?

>  > What exactly was "removed"?  The variable
>  > 'open-paren-in-column-0-is-defun-start' seems to still exist, and
>  > its default value seems to be t (my .emacs does not set it and it
>  > is t).


>  >	> With the argument that people want to comment out larger
>  >	> regions of code by simply putting comment delimiters at the
>  >	> beginning and end of that code.

>  > You have not said concretely which practice this refers to, but the
>  > feature of special meaning for open-paren in column zero affects Emacs
>  > editing commands, not the meaning of the program when executed.

> 'open-paren-in-column-0-is-defun-start', if non-nil, conceptually allows
> progmodes to avoid scanning an entire buffer in order to get things like
> syntax highlighting and code indenting right.  Rather, progmodes are
> allowed to find the first or next paren in column zero wrt a given
> position and base further decisions on the assumption that such a paren
> is on the "top level" of its buffer.

This assumption proved to be very problematic.  The fact is, people put
parentheses in column zero inside comments, and nothing we can say or do
will stop them.  Why should it?  these parentheses are perfectly valid
in so many languages.  Most notoriously was bug #22884, where there were
such parentheses in Emacs's own C sources.  And there were quite a lot
of ostensible bugs in CC Mode caused by these parentheses.

> Recent Emacsen either ignore that variable or silently reset it to nil
> internally so it doesn't get into their way.

The one place which matters where o-p-i-c-0-i-d-s used to be used was in
back_comment() in syntax.c.  This was removed for the above reasons.

> Their progmodes either always scan an entire buffer from its beginning
> or use some elaborate, fragile techniques to find such a top level
> position.

Why do you use the word "fragile" here?  Do you have examples of this
code failing?  It seems to me the current way is somewhat less fragile
that the o-p-i-c-0-i-d-s way.

> Moreover, our underlying mechanism for syntax highlighting always
> marks the entire rest of a buffer as dirty after every single editing
> change.

I have always been sceptical of the wisdom of this.  Why invalidate the
entire rest of the buffer, when a typical buffer change will cause at
most local changes to the fontification?  I think this can only be for
ease of coding in the font-lock functions.

Another approach would be somehow to divide a buffer into cells,
something like one cell per function.  A buffer change would then
invalidate the fontification only to the end of the current cell, not
EOB.  This would be faster, but somewhat complicated to implement.

> This has the consequence that that entire part has to be continuously
> rescanned when some of it is shown in another window.

> The basic slowness of Emacs over the past years is a direct consequence
> of that policy.  Most people don't care because they are using fast
> processors that easily compensate the resulting overhead.  People with
> old and slow hardware suffer from it.

How bad is this on your machine, really?

> So since you earlier asked for "a switch to turn off the changes" then
> my answer is that such a switch already exists but has been deactivated.

As already stated, it was deactivated for good reasons.  If we were to
bring it back, I think we would need to add heuristics around
paren-in-column-0 detection to the former rather crass 100% quitting of
back_comment when one is encountered.  I've considered this quite a bit
in the past, and can't conceive of anything both helpful and fast.

> martin

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Paul Eggert]

On 4/2/20 11:59 PM, martin rudalics wrote:
 > The basic slowness of Emacs over the past years is a direct consequence
 > of that policy.

How many users is that true for? When Emacs is slow for me, it's usually because 
of very long lines. Is that the same issue, or a different one?

On 4/3/20 9:23 AM, martin rudalics wrote:
>  >          Stefan "whose fastest machine has an Intel i3-4170"
> 
> martin "on an AMD Athlon 64 X2 Dual Core Processor 5000+"

Is that the Black Edition 5000+ or the regular one? The Black Edition was quite 
the thing in 2007. :-)

-- Paul "on an AMD Phenom II X4 910e" circa 2010

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > I understand proving/disproving it is hard.

We've been through this before.  What made me stumble over it this time
is Bug#40278 where I was not able to reproduce any flickering with the
OP's scenario because here scrolling buffers like dispextern.h and
xdisp.c takes some 10 seconds to complete.  I did not investigate what
precisely makes scrolling so slow.  The shear fact that this happens
exclusively with C buffers is enough evidence for me to point at CC mode
as the major culprit.

Now if we had left in 'open-paren-in-column-0-is-defun-start' and would
still be able to handle its non-nil value in some rudimentary way, with
all the bugs it may cause, we would have at least some evidence at our
hands to say, for example, that "going beyond some open paren is
dangerous in the case at hand" and might allow us to spot some
unrestricted movement in that direction.  But as it is, we have no clue,
no smoking gun, and every evidence about where cc-mode moves to when
analyzing the construct at point must be (re-)constructed in a rather
cumbersome way.

 > I'm really asking what are the signs that make you think it's
 > due to the lack of use of `open-paren-in-column-0-is-defun-start`.

We know that parsing a C buffer from the beginning, using something like
our 'parse-partial-sexp', is straightforward and cannot be responsible
for the scrolling problem I mention above.  So the cause must be in a
backtracking step performed by CC mode, either because something in the
state established by parsing does not allow to fully identify the
construct at where parsing stopped (unlikely IMO) or because scrolling
the buffer did overwrite some important part of the state established by
an earlier parsing step.  Note that I'm only talking about scrolling
such buffers, no buffer modifications that would invalidate an earlier
established state are involved here.

 >> But customizing 'open-paren-in-column-0-is-defun-start' has no effect
 >> here any more and reconstructing its earlier semantics would consume
 >> more resources than I can afford.
 >
 > Yes, I don't think there's much interested in going back to that.
 > OTOH if we could find specific cases where the old code was faster
 > (supposedly due to its use of `open-paren-in-column-0-is-defun-start`),
 > we could think about how to recover that performance.

'open-paren-in-column-0-is-defun-start' was dismounted in many steps
spread over many years.  We know that we can't go back there.

 > E.g. `open-paren-in-column-0-is-defun-start` was used in
 > `forward-comment` was scanning comments backward to avoid having to use
 > `parse-partial-sexp` all the way from `point-min` in the worst case.
 > We replaced that with a parse cache (`syntax-ppss`) which seems to be
 > just as good in my experience (and significantly better in those
 > languages where open parens in column 0 are virtually non-existent).

This goes without saying and is also a good argument in the case of C
header files like dispextern.h where the missing
'open-paren-in-column-0-is-defun-start' can be hardly the cause of the
poor performance.

 > If we can show that this replacement is not "just as good", there are
 > various options to improve it or replace it, but we'd need to know which
 > case(s) matter in order to design it.

'open-paren-in-column-0-is-defun-start' would be one knob to press when
things go wrong in some untraceable way.  It would provide a way to
narrow the current buffer down to the three nearest functions around
point and forget about the rest.  As it is, it has no effect at all.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >  > The basic slowness of Emacs over the past years is a direct consequence
 >  > of that policy.
 >
 > How many users is that true for? When Emacs is slow for me, it's usually because of very long lines. Is that the same issue, or a different one?

As mentioned in my answer to Stefan the most recent issue I noticed is
that mouse-wheel scrolling buffers for xdisp.c and dispextern.h consumes
100% of my CPU and still takes some ten seconds to complete.

 > Is that the Black Edition 5000+ or the regular one? The Black Edition was quite the thing in 2007. :-)

How would I find out?

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > This assumption proved to be very problematic.  The fact is, people put
 > parentheses in column zero inside comments, and nothing we can say or do
 > will stop them.  Why should it?  these parentheses are perfectly valid
 > in so many languages.  Most notoriously was bug #22884, where there were
 > such parentheses in Emacs's own C sources.  And there were quite a lot
 > of ostensible bugs in CC Mode caused by these parentheses.

All these problems could have been cured easily: People who want such
parentheses would have set 'open-paren-in-column-0-is-defun-start' to
nil (or used a default value of nil) and everybody would have been
happy.  But throwing out the child with the bathwater by eliminating the
effect of that variable completely has left us no clue as to what makes
scrolling so slow in the present situation.

 >> Recent Emacsen either ignore that variable or silently reset it to nil
 >> internally so it doesn't get into their way.
 >
 > The one place which matters where o-p-i-c-0-i-d-s used to be used was in
 > back_comment() in syntax.c.  This was removed for the above reasons.

IIRC that was just the last nail in the variable's coffin.

 >> Their progmodes either always scan an entire buffer from its beginning
 >> or use some elaborate, fragile techniques to find such a top level
 >> position.
 >
 > Why do you use the word "fragile" here?  Do you have examples of this
 > code failing?  It seems to me the current way is somewhat less fragile
 > that the o-p-i-c-0-i-d-s way.

For me it's fragile because it does not allow me to easily hack it.  I
once gave up after trying in vain to implement a very elementary thing:
Make adjusting the first paragraph of the doc-string of an Elisp routine
not add anything to the first line of that doc-string and thus
invalidate our rules for writing doc-strings.  As things stand, I still
manually insert an empty line after the first one, adjust my paragraph,
and remove that empty line afterwards.

 >> Moreover, our underlying mechanism for syntax highlighting always
 >> marks the entire rest of a buffer as dirty after every single editing
 >> change.
 >
 > I have always been sceptical of the wisdom of this.  Why invalidate the
 > entire rest of the buffer, when a typical buffer change will cause at
 > most local changes to the fontification?  I think this can only be for
 > ease of coding in the font-lock functions.

I recall that more than a decade ago I tried to convince Stefan that a
buffer change that leaves the ppss unchanged should not invalidate the
rest of the buffer.  He initially liked the idea but didn't want to
implement it.

 > Another approach would be somehow to divide a buffer into cells,
 > something like one cell per function.  A buffer change would then
 > invalidate the fontification only to the end of the current cell, not
 > EOB.  This would be faster, but somewhat complicated to implement.

Why?

 > How bad is this on your machine, really?

I already cited the scrolling example in my other posts.  Scrolling
dispextern.h and xdisp.c each take 10 seconds for one turn of the wheel
and consume 100% of the allotted CPU here.

 >> So since you earlier asked for "a switch to turn off the changes" then
 >> my answer is that such a switch already exists but has been deactivated.
 >
 > As already stated, it was deactivated for good reasons.  If we were to
 > bring it back, I think we would need to add heuristics around
 > paren-in-column-0 detection to the former rather crass 100% quitting of
 > back_comment when one is encountered.  I've considered this quite a bit
 > in the past, and can't conceive of anything both helpful and fast.

I still don't understand why it had to be eliminated.  Defaulting it to
nil but respecting a non-nil value would have been completely sufficient
IMHO.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Martin.

On Sat, Apr 04, 2020 at 10:56:27 +0200, martin rudalics wrote:
>  > This assumption proved to be very problematic.  The fact is, people put
>  > parentheses in column zero inside comments, and nothing we can say or do
>  > will stop them.  Why should it?  these parentheses are perfectly valid
>  > in so many languages.  Most notoriously was bug #22884, where there were
>  > such parentheses in Emacs's own C sources.  And there were quite a lot
>  > of ostensible bugs in CC Mode caused by these parentheses.

> All these problems could have been cured easily: People who want such
> parentheses would have set 'open-paren-in-column-0-is-defun-start' to
> nil (or used a default value of nil) and everybody would have been
> happy.  But throwing out the child with the bathwater by eliminating the
> effect of that variable completely has left us no clue as to what makes
> scrolling so slow in the present situation.

Well, there aren't "people who want such parentheses".  They just
happen, without any conscious decision on the user's part.  Maybe, as
you say, changing o-p-i-c-0-i-d-s's default to nil would have been better.

But I'm a bit puzzled by the "scrolling so slow" bit.

[ .... ]

(I hope to get around to commenting on the rest of your post, sometime.)

> I already cited the scrolling example in my other posts.  Scrolling
> dispextern.h and xdisp.c each take 10 seconds for one turn of the wheel
> and consume 100% of the allotted CPU here.

There's something very strange going on, here.  If I do

    (time-it (scroll-up) (sit-for 0))

in a region of xdisp.c where the next screenfull hasn't yet been
fontified, it takes 0.029s on my machine.  For you it's taking 10s.
That's a factor of ~350.  Even taking into account me running on an
optimised build, and you (presumably) on a debug build, that gives a
factor of ~100 difference.

I appreciate you needing to run in a debug build, but what sort of
timing do you get on an optimised build?

I think you said you have a state of the art 2007 processor.  Mine is
from 2017.  Processing power did not speed up by a factor of 100 in that
period.

There is something very strange happening here.

[ .... ]

> I still don't understand why it
> [open-paren-in-column-0-is-defun-start] had to be eliminated.
> Defaulting it to nil but respecting a non-nil value would have been
> completely sufficient IMHO.

Maybe we didn't have your benchmark at the time the change was made,
which I think was around 2015.

> martin

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > (time-it (scroll-up) (sit-for 0))

To avoid confusion - what is 'time-it'?  And please note that I did not
say that I ran 'scroll-up'.  I did a mouse wheel scroll as the OP of
that bug requested and unfortunately I don't know what mouse wheel
scrolling does here because C-h k on it just gives me

Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil)
   help--read-key-sequence()
   byte-code("\300 C\207" [help--read-key-sequence] 1)
   call-interactively(describe-key nil nil)
   command-execute(describe-key)

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: eliz@gnu.org, rrandresf@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Sat, 4 Apr 2020 13:10:43 +0200
> 
>  > (time-it (scroll-up) (sit-for 0))
> 
> To avoid confusion - what is 'time-it'?  And please note that I did not
> say that I ran 'scroll-up'.  I did a mouse wheel scroll as the OP of
> that bug requested and unfortunately I don't know what mouse wheel
> scrolling does here because C-h k on it just gives me
> 
> Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil)
>    help--read-key-sequence()
>    byte-code("\300 C\207" [help--read-key-sequence] 1)
>    call-interactively(describe-key nil nil)
>    command-execute(describe-key)

It runs mwheel-scroll.  Not sure why it signals an error on your
system.

And if I turn the mouse wheel just one click, I don't see Emacs
lagging behind, either.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Martin.

On Sat, Apr 04, 2020 at 13:10:43 +0200, martin rudalics wrote:
>  > (time-it (scroll-up) (sit-for 0))

> To avoid confusion - what is 'time-it'?

    (defmacro time-it (&rest forms)
      "Time the running of a sequence of forms using `float-time'.
    Call like this: \"M-: (time-it (foo ...) (bar ...) ...)\"."
      `(let ((start (float-time)))
        ,@forms
        (- (float-time) start)))

> And please note that I did not say that I ran 'scroll-up'.  I did a
> mouse wheel scroll as the OP of that bug requested ....

Sorry.  I thought you were using poetic language when you wrote "one turn
of the wheel".  I didn't have the context of the bug in my head.

Just for comparison, would you please time (scroll-up) (sit-for 0) in
xdisp.c on your machine, to give us a better sense of the relative
speeds.

For what it's worth, I've just tried a fast mouse-wheel scroll in xdisp.c
in X Windows on my machine.  There was no noticeable lag.

> .... and unfortunately I don't know what mouse wheel scrolling does
> here because C-h k on it just gives me

> Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil)
>    help--read-key-sequence()
>    byte-code("\300 C\207" [help--read-key-sequence] 1)
>    call-interactively(describe-key nil nil)
>    command-execute(describe-key)

This doesn't happen to me.  I get the information about mwheel-scroll.

> martin

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello again, Martin.

On Sat, Apr 04, 2020 at 10:56:27 +0200, martin rudalics wrote:
>  > This assumption proved to be very problematic.  The fact is, people put
>  > parentheses in column zero inside comments, and nothing we can say or do
>  > will stop them.  Why should it?  these parentheses are perfectly valid
>  > in so many languages.  Most notoriously was bug #22884, where there were
>  > such parentheses in Emacs's own C sources.  And there were quite a lot
>  > of ostensible bugs in CC Mode caused by these parentheses.

> All these problems could have been cured easily: People who want such
> parentheses would have set 'open-paren-in-column-0-is-defun-start' to
> nil (or used a default value of nil) and everybody would have been
> happy.  But throwing out the child with the bathwater by eliminating the
> effect of that variable completely has left us no clue as to what makes
> scrolling so slow in the present situation.

>  >> Recent Emacsen either ignore that variable or silently reset it to nil
>  >> internally so it doesn't get into their way.

>  > The one place which matters where o-p-i-c-0-i-d-s used to be used was in
>  > back_comment() in syntax.c.  This was removed for the above reasons.

> IIRC that was just the last nail in the variable's coffin.

My impression was that this was the single place in the syntax routines
it was ever used, though back_comment is called from several places in
syntax.c.

[ .... ]

>  >> Moreover, our underlying mechanism for syntax highlighting always
>  >> marks the entire rest of a buffer as dirty after every single
>  >> editing change.

>  > I have always been sceptical of the wisdom of this.  Why invalidate
>  > the entire rest of the buffer, when a typical buffer change will
>  > cause at most local changes to the fontification?  I think this can
>  > only be for ease of coding in the font-lock functions.

> I recall that more than a decade ago I tried to convince Stefan that a
> buffer change that leaves the ppss unchanged should not invalidate the
> rest of the buffer.  He initially liked the idea but didn't want to
> implement it.

That sounds like a shame.

>  > Another approach would be somehow to divide a buffer into cells,
>  > something like one cell per function.  A buffer change would then
>  > invalidate the fontification only to the end of the current cell,
>  > not EOB.  This would be faster, but somewhat complicated to
>  > implement.

> Why?

Because we'd need somehow to detect when the cell boundaries change,
possibly coalescing two cells, possibly creating a new one.  This would
involve delicate coding in an after-change-function, possibly in a
before-c-f too.

We'd need to avoid prematurely coalescing two functions (for example,
when typing in an open paren in emacs-lisp-mode).

Also, new data structures would be needed in font-lock.

I'm not saying it couldn't be done, but it would take some effort, and
might be politically difficult, too.

[ .... ]

>  >> So since you earlier asked for "a switch to turn off the changes"
>  >> then my answer is that such a switch already exists but has been
>  >> deactivated.

>  > As already stated, it was deactivated for good reasons.  If we were
>  > to bring it back, I think we would need to add heuristics around
>  > paren-in-column-0 detection to the former rather crass 100% quitting
>  > of back_comment when one is encountered.  I've considered this quite
>  > a bit in the past, and can't conceive of anything both helpful and
>  > fast.

Maybe the following would do it: on every open paren at column 0 within a
string or comment, apply a text property called `syntax-not-defun-start'.
back_comment would skip past any such paren with that property.  It would
be the responsibility of the major mode code to add and remove the
property.  Again, this would mean a moderately intricate
after-change-function, possibly paired with a before-change-function.

This would perhaps remove the trouble from o-p-i-c-0-i-d-s and allow us
to put it back, but I'm not sure Stefan M. would like it.

> I still don't understand why it had to be eliminated.  Defaulting it to
> nil but respecting a non-nil value would have been completely sufficient
> IMHO.

I think you might well be right, there.

> martin

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> I still don't understand why it had to be eliminated.  Defaulting it to
> nil but respecting a non-nil value would have been completely sufficient
> IMHO.

I can answer why it wasn't changed to nil:

1- o-p-i-c-0-i-d-s had an "invisible" effect on `forward-comment`, where
   the only visible effect was that sometimes it caused
   `forward-comment` to misbehave.  This use was just an optimization.
   Setting the var to nil avoided the misbehavior but could result in
   *much* worse performance in some cases (e.g. calling
   `parse-partial-sexp` from `point-min` once per line skipped when
   scanning text backward, so O(N*M) where N is the size of the buffer
   and M is the distance we're moving).  So just changing the var to nil
   was really not an option, it needed to be replaced by *another*
   optimization.  BTW, I still see not reason to link this other
   optimization, which relies on `syntax-ppss` with the scrolling
   performance problems you are seeing, especially if it's only visible
   in CC-mode buffers and not in things like Javascript (which have an
   almost identical syntax-table).

2- o-p-i-c-0-i-d-s also had (and still has) an effect on
   `beginning-of-defun` and this effect is supposed to be visible and
   predictable.  It's not just an optimization.  So changing the var to
   nil would have changed the behavior of `beginning-of-defun` and there
   was a very clear resistance to that.  Another reason not to just
   change its value to nil.

I also can't imagine how defaulting to nil "would have been completely
sufficient IMHO": setting the var to nil had been an option for decades
but it never enjoyed much popularity.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1378 bytes --]

On 4/4/20 1:55 AM, martin rudalics wrote:
>  >  > The basic slowness of Emacs over the past years is a direct consequence
>  >  > of that policy.
>  >
>  > How many users is that true for? When Emacs is slow for me, it's usually 
> because of very long lines. Is that the same issue, or a different one?
> 
> As mentioned in my answer to Stefan the most recent issue I noticed is
> that mouse-wheel scrolling buffers for xdisp.c and dispextern.h consumes
> 100% of my CPU and still takes some ten seconds to complete.

I looked into this, and although it's no doubt fundamentally due to a slow 
algorithm, the slowness is exacerbated if you use -Og (which you appear to be 
using). Stefan's recent message hinted at this. I installed the attached into 
master to try to fix the -Og issue; please give it a try.

The slow algorithm should be fixed too, but I'm no expert there.

>  > Is that the Black Edition 5000+ or the regular one? The Black Edition was 
> quite the thing in 2007. :-)
> 
> How would I find out?

Your BIOS right after cycling power, I expect. It's not high priority to find 
out. As I vaguely recall the main advantage of the Black Edition is that you 
could overclock, and if you had a Black Edition my next suggestion was going to 
be a joke that you should overclock your ancient and slow CPU to make your Emacs 
faster....

[-- Attachment #2: gcc-Og-patch.diff --]
[-- Type: text/x-patch, Size: 2331 bytes --]

diff --git a/src/Makefile.in b/src/Makefile.in
index 552dd2e50a..dfd322553b 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -377,11 +377,14 @@ pdmp :=
 # Flags that might be in WARN_CFLAGS but are not valid for Objective C.
 NON_OBJC_CFLAGS = -Wignored-attributes -Wignored-qualifiers -Wopenmp-simd

+# Cajole GCC into inlining key ops even if it wouldn't normally.
+KEY_OPS_CFLAGS = $(if $(filter -Og,$(CFLAGS)),-DDEFINE_KEY_OPS_AS_MACROS)
+
 # -Demacs makes some files produce the correct version for use in Emacs.
 # MYCPPFLAGS is for by-hand Emacs-specific overrides, e.g.,
 # "make MYCPPFLAGS='-DDBUS_DEBUG'".
-EMACS_CFLAGS=-Demacs $(MYCPPFLAGS) -I. -I$(srcdir) \
-  -I$(lib) -I$(top_srcdir)/lib \
+EMACS_CFLAGS = -Demacs $(KEY_OPS_CFLAGS) $(MYCPPFLAGS) \
+  -I. -I$(srcdir) -I$(lib) -I$(top_srcdir)/lib \
   $(C_SWITCH_MACHINE) $(C_SWITCH_SYSTEM) $(C_SWITCH_X_SITE) \
   $(GNUSTEP_CFLAGS) $(CFLAGS_SOUND) $(RSVG_CFLAGS) $(IMAGEMAGICK_CFLAGS) \
   $(PNG_CFLAGS) $(LIBXML2_CFLAGS) $(DBUS_CFLAGS) \
diff --git a/src/lisp.h b/src/lisp.h
index f223814d8f..7fc3af992e 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -411,15 +411,19 @@ #define lisp_h_XHASH(a) XUFIXNUM_RAW (a)
 # define lisp_h_XTYPE(a) ((enum Lisp_Type) (XLI (a) & ~VALMASK))
 #endif

-/* When compiling via gcc -O0, define the key operations as macros, as
-   Emacs is too slow otherwise.  To disable this optimization, compile
-   with -DINLINING=false.  */
-#if (defined __NO_INLINE__ \
-     && ! defined __OPTIMIZE__ && ! defined __OPTIMIZE_SIZE__ \
-     && ! (defined INLINING && ! INLINING))
-# define DEFINE_KEY_OPS_AS_MACROS true
-#else
-# define DEFINE_KEY_OPS_AS_MACROS false
+/* When DEFINE_KEY_OPS_AS_MACROS, define key operations as macros to
+   cajole the compiler into inlining them; otherwise define them as
+   inline functions as this is cleaner and can be more efficient.
+   The default is true if the compiler is GCC-like and if function
+   inlining is disabled because the compiler is not optimizing or is
+   optimizing for size.  Otherwise the default is false.  */
+#ifndef DEFINE_KEY_OPS_AS_MACROS
+# if (defined __NO_INLINE__ \
+      && ! defined __OPTIMIZE__ && ! defined __OPTIMIZE_SIZE__)
+#  define DEFINE_KEY_OPS_AS_MACROS true
+# else
+#  define DEFINE_KEY_OPS_AS_MACROS false
+# endif
 #endif

 #if DEFINE_KEY_OPS_AS_MACROS

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

Hi Paul,

On 05.04.2020 03:07, Paul Eggert wrote:
> I looked into this, and although it's no doubt fundamentally due to a 
> slow algorithm, the slowness is exacerbated if you use -Og (which you 
> appear to be using). Stefan's recent message hinted at this. I installed 
> the attached into master to try to fix the -Og issue; please give it a try.

I seem to be getting a 2x perf improvement from this. Which is not bad.

Speaking of the patch, you might want to update the "Maybe in the year 
2020" sentence as well (again). :-)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > 'open-paren-in-column-0-is-defun-start', if non-nil, conceptually allows
  > > progmodes to avoid scanning an entire buffer in order to get things like
  > > syntax highlighting and code indenting right.  Rather, progmodes are
  > > allowed to find the first or next paren in column zero wrt a given
  > > position and base further decisions on the assumption that such a paren
  > > is on the "top level" of its buffer.

  > This assumption proved to be very problematic.

Problematic for whom?  I don't see why.

						    The fact is, people put
  > parentheses in column zero inside comments, and nothing we can say or do
  > will stop them.  Why should it?

Why would we want to "stop them"?  That feature is/was a _convenience_
for users, a speedup.  If you don't appreciate the speedup, so you decide
not to arrange to get it, why should I argue with you?  I can still benefit
from that speedup in my projects. so I am happy and so are you.

				     these parentheses are perfectly valid
  > in so many languages.

Nobody said they were "invalid" in any language.  I think we are
miscommunicating.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >> ...  I don't know what mouse wheel
 >> scrolling does here because C-h k on it just gives me
 >>
 >> Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil)
 >>     help--read-key-sequence()
 >>     byte-code("\300 C\207" [help--read-key-sequence] 1)
 >>     call-interactively(describe-key nil nil)
 >>     command-execute(describe-key)
 >
 > It runs mwheel-scroll.  Not sure why it signals an error on your
 > system.

I see.  C-h k apparently doesn't work with my separate minibuffer frame
setup.  Maybe some snafu with 'mouse-wheel--get-scroll-window'.

 > And if I turn the mouse wheel just one click, I don't see Emacs
 > lagging behind, either.

Right.  The effect appears only after a number of turns.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >> To avoid confusion - what is 'time-it'?
 >
 >      (defmacro time-it (&rest forms)
 >        "Time the running of a sequence of forms using `float-time'.
 >      Call like this: \"M-: (time-it (foo ...) (bar ...) ...)\"."
 >        `(let ((start (float-time)))
 >          ,@forms
 >          (- (float-time) start)))

Thanks.  But for 'mwheel-scroll' I would have to construct an EVENT
first.  I'm not yet sure how to that.

 >> And please note that I did not say that I ran 'scroll-up'.  I did a
 >> mouse wheel scroll as the OP of that bug requested ....
 >
 > Sorry.  I thought you were using poetic language when you wrote "one turn
 > of the wheel".  I didn't have the context of the bug in my head.

Please have a look at Bug#40278 which is about mouse wheel scrolling, a
child frame, Windows 10 and dispextern.h.  People in that thread mainly
concentrated on the child frame but here on Debian all I needed to clog
my machine was to display dispextern.h and spin the mouse wheel.

 > Just for comparison, would you please time (scroll-up) (sit-for 0) in
 > xdisp.c on your machine, to give us a better sense of the relative
 > speeds.

With your 'time-it' it gives me 0.32201647758483887 here (seconds?).  I
have not managed yet to put scroll-up's into a loop and redisplay after
each call so I could simulate/time the effect of mouse wheel rolling.

 > For what it's worth, I've just tried a fast mouse-wheel scroll in xdisp.c
 > in X Windows on my machine.  There was no noticeable lag.

"fast" is probably not sufficient.  Here a few rolls suffice to freeze
the Emacs frame for about 10 secs.  Maybe on your machine you need more
rolls.  Try to roll fast until you reach the end of the buffer.

I meanwhile suspect that one culprit might be the progressive squaring
in 'mwheel-scroll'.  But I don't understand that yet and do not see it
documented well.  Also, setting 'mouse-wheel-scroll-amount' to '(1) does
not seem to make any difference either.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > I looked into this, and although it's no doubt fundamentally due to a
 > slow algorithm, the slowness is exacerbated if you use -Og (which you
 > appear to be using).

I'm using '-O0 -g3 -no-pie' for builds that I use to debug issues and
'-O3 -no-pie' for builds that I use for daily work.  If I were to change
this (involving some 50 builds on various machines and partitions) I
would need to be convinced that the new options would not change
anything when running the debug builds and would not slow down anything
when running the optimized builds.

 > Stefan's recent message hinted at this. I
 > installed the attached into master to try to fix the -Og issue; please
 > give it a try.

 > The slow algorithm should be fixed too, but I'm no expert there.
 >
 >>  > Is that the Black Edition 5000+ or the regular one? The Black Edition was quite the thing in 2007. :-)
 >>
 >> How would I find out?

 > Your BIOS right after cycling power, I expect. It's not high priority
 > to find out. As I vaguely recall the main advantage of the Black
 > Edition is that you could overclock, and if you had a Black Edition my
 > next suggestion was going to be a joke that you should overclock your
 > ancient and slow CPU to make your Emacs faster....

It offers me an overclock mode with a CPU frequency of 200 MHz and a
PCIE Frequency of 100.  I also have M2 Boost disabled whatever that
means.  Still no idea whether that means that I have a Black Edition.
Maybe I'll install a program called CPU-Z (or CPUID ?) to find out.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Richard.

On Sat, Apr 04, 2020 at 23:12:49 -0400, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

>   > > 'open-paren-in-column-0-is-defun-start', if non-nil,
>   > > conceptually allows progmodes to avoid scanning an entire buffer
>   > > in order to get things like syntax highlighting and code
>   > > indenting right.  Rather, progmodes are allowed to find the
>   > > first or next paren in column zero wrt a given position and base
>   > > further decisions on the assumption that such a paren is on the
>   > > "top level" of its buffer.

>   > This assumption proved to be very problematic.

> Problematic for whom?

Well, for me for a start.  ;-)  I had to deal with lots of bug reports,
the solution to which was "please put a \ before the ( in your comment".
One of these was in the copyright statement in the Emacs C sources (see
bug #22884).  This bug caused the typing of a "//" to take over 10
seconds to redisplay.

> I don't see why.

Suppose open-paren-in-column-0-is-defun-start is non-nil, and we have
something like:

    {
    /* foo
    (but bar
    */
     }
    ^
  point

.  From the indicated point, do (scan-lists (point) -1 1).

(i) scan-lists moves back and finds the comment end....
(ii) ... and thus calls back_comment.
(iii) back_comment finds the ( in column zero, and wrongly assumes this
  is the beginning of a function.
(iv) back_comment breaks off its scanning, returning "comment start not
  found".
(v) scan_lists carries on the scanning from just before the "*/", now
  blissfully unaware it's actually inside a comment.
(vi) scan_lists finds the (, and returns its position.

The indentation and/or fontification functions in CC Mode are now
confused.

>   > The fact is, people put parentheses in column zero inside
>   > comments, and nothing we can say or do will stop them.  Why should
>   > it?

> Why would we want to "stop them"?  That feature is/was a _convenience_
> for users, a speedup.  If you don't appreciate the speedup, so you decide
> not to arrange to get it, why should I argue with you?  I can still benefit
> from that speedup in my projects. so I am happy and so are you.

o-p-i-c-0-i-d-s is an arcane variable which very few users are aware of.
Their first knowledge of it typically came in my response to their bug
report saying put the backslash before the paren.  There are likely many
more users who have encountered wrong indentation/fontification who have
just not bothered reporting it.

>   > these parentheses are perfectly valid in so many languages.

> Nobody said they were "invalid" in any language.  I think we are
> miscommunicating.

I think I meant that, given their validity, it is up to us as Emacs
developers to arrange that they don't cause trouble, rather than
expecting our users to insert these obtrusive backslashes.

> -- 
> Dr Richard Stallman
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rrandresf@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Sun, 5 Apr 2020 10:44:24 +0200
> 
>  > And if I turn the mouse wheel just one click, I don't see Emacs
>  > lagging behind, either.
> 
> Right.  The effect appears only after a number of turns.

Can you tell how many times you turn the wheel until Emacs gets stuck?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Martin.

On Fri, Apr 03, 2020 at 08:59:54 +0200, martin rudalics wrote:

[ .... ]

> 'open-paren-in-column-0-is-defun-start', if non-nil, conceptually allows
> progmodes to avoid scanning an entire buffer in order to get things like
> syntax highlighting and code indenting right.  Rather, progmodes are
> allowed to find the first or next paren in column zero wrt a given
> position and base further decisions on the assumption that such a paren
> is on the "top level" of its buffer.

> Recent Emacsen either ignore that variable or silently reset it to nil
> internally so it doesn't get into their way.

Correction to my recent posts here.  o-p-i-c-0-i-d-s still exists and
can still work.

Currently, its operation is blocked by the variable
comment-use-syntax-ppss being non-nil.  I'm not sure what this variable
is for (it was introduced in:

    commit 14b95587520959c5b54356547a0a69932a9bb480
    Author: Stefan Monnier <monnier@iro.umontreal.ca>
    Date:   Tue Dec 12 23:03:00 2017 -0500

        * src/syntax.c (find_defun_start): Use syntax-ppss

        (syms_of_syntax): New variable comment-use-syntax-ppss.

), but setting it to nil does indeed reenable the o-p-i-c-0-i-d-s
behaviour.  :-)

[ .... ]

> So since you earlier asked for "a switch to turn off the changes" then
> my answer is that such a switch already exists but has been deactivated.

Please try setting comment-use-syntax-ppss to nil.

> martin

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> I think I meant that, given their validity, it is up to us as Emacs
> developers to arrange that they don't cause trouble, rather than
> expecting our users to insert these obtrusive backslashes.

Indeed.  And the reverse is true as well: while it might be OK for
a user to put his open-parens in column 0 to help Emacs be more
responsive, it's not OK for Emacs to be too slow on files where there
are no such open-parens in column 0.  So Emacs has to use some other
mechanism anyway to avoid the slowdown in such files.  Once you have
this mechanism in place, the "open-parens in column 0" heuristic is just
not worth the trouble any more.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > Can you tell how many times you turn the wheel until Emacs gets stuck?

Five or six times where a "turn of the wheel" means to put the finger at
one end of the opening of the wheel and turn the wheel until the finger
is at the opposite end, approximately.  Each such turn seems to provoke
between 5 and 7 subsequent calls of 'mwheel-scroll' here so it
apparently takes 30 to 35 calls to get into those ~10 second freezes.
33 is a typical number I've seen, using a simple counter within
'mwheel-scroll'.  A typical value of the cumulative number of lines to
scroll (AMT in 'mwheel-scroll') then seems to be about 3000.  I would
have to run this more often to get better values.

The first five or so of such turns are seemingly processed normally.
After that, input apparently arrives faster than Emacs is able to
process it and no redisplay takes place until all input has been
consumed.

I cannot nearly simulate the problem using a simple loop that steadily
invokes say (forward-line 50) followed by a (sit-for 0).  50 such
invocations take 16.7 seconds here, 22.7 with (forward-line -50).  And
redisplay takes place as expected here.  So it's likely the mouse wheel
input mounting up that inhibits redisplay.

Do you at least see CPU activity significantly go up when you do such
mouse wheel scrolling?  I rarely use mouse wheel scrolling in Emacs so I
cannot say what is typical and what isn't.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > Please try setting comment-use-syntax-ppss to nil.

This has zero impact here.  Even when going to the end of the buffer
first and scrolling backward afterwards.  I have no idea yet what's
going on here and unless someone else can confirm my observations (Noam
was the only person so far who at least noticed that

   I can see it here, especially on scroll towards the beginning of the
   file (Emacs is maxing out the CPU while doing this).

but apparently nobody else has seen it) I have to dig into this myself.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rrandresf@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Sun, 5 Apr 2020 17:50:33 +0200
> 
> I cannot nearly simulate the problem using a simple loop that steadily
> invokes say (forward-line 50) followed by a (sit-for 0).  50 such
> invocations take 16.7 seconds here, 22.7 with (forward-line -50).  And
> redisplay takes place as expected here.  So it's likely the mouse wheel
> input mounting up that inhibits redisplay.

Not forward-line, scroll-up.  That's what wheel.el calls.

> Do you at least see CPU activity significantly go up when you do such
> mouse wheel scrolling?

With what build?  With the -O0 build with --enable-checking, I don't
need the mouse: it's enough to lean on C-v and let the keyboard
auto-repeat do its job -- one execution unit of the CPU maxes out
after 5 to 10 C-v's.

But with a -O2 optimized production build, Emacs keeps up both when I
lean on C-v and when I turn the mouse wheel constantly.

Anyway, you don't need to work too hard to get me started on how slow
CC Mode is, especially when re-indentation is involved in addition to
font-lock.  Which is why I think we should move away of regexp-based
and SMIE-based engines towards fast parsers written in C.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 05.04.2020 19:15, Eli Zaretskii wrote:
>> Do you at least see CPU activity significantly go up when you do such
>> mouse wheel scrolling?
> With what build?  With the -O0 build with --enable-checking, I don't
> need the mouse: it's enough to lean on C-v and let the keyboard
> auto-repeat do its job -- one execution unit of the CPU maxes out
> after 5 to 10 C-v's.
> 
> But with a -O2 optimized production build, Emacs keeps up both when I
> lean on C-v and when I turn the mouse wheel constantly.

Same here. And much as I like to see certain modes improved, I'm not 
sure this is only a performance problem. Because no matter how fast 
syntax highlighting is, you can lean on C-v faster.

IIRC, redisplay-dont-pause was supposed to help with things like this. 
But it doesn't (and it's obsolete), and apparently jit-lock does its 
thing anyway. This looks like the interesting part of the profile:

     - scroll-up                                              5657  73%
      - jit-lock-function                                     5444  70%
       - jit-lock-fontify-now                                 5439  70%
        + jit-lock--run-functions                             5431  70%
        + run-with-timer                                         5   0%

Do you guys want to try (setq jit-lock-defer-time 0)? It comes with a 
certain visual downside, though.

Or, alternatively, (setq fast-but-imprecise-scrolling t). This var seems 
like a good idea in general, so we might consider going further with it.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rrandresf@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 5 Apr 2020 21:05:00 +0300
> 
> Do you guys want to try (setq jit-lock-defer-time 0)? It comes with a 
> certain visual downside, though.
> 
> Or, alternatively, (setq fast-but-imprecise-scrolling t). This var seems 
> like a good idea in general, so we might consider going further with it.

No, I want fontification to work fast enough to not require these
measures.  They are retreat of a kind: we are giving up correctness
because we are annoyed by slowness.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 05.04.2020 21:16, Eli Zaretskii wrote:
> No, I want fontification to work fast enough to not require these
> measures.  They are retreat of a kind: we are giving up correctness
> because we are annoyed by slowness.

But what's "fast enough"? Fontification that is faster than the signal 
rate of the keyboard?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 5 Apr 2020 21:05:00 +0300
> Cc: acm@muc.de, rrandresf@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> 
> IIRC, redisplay-dont-pause was supposed to help with things like this. 

Sorry, missed that part.

No, redisplay-dont-pause wasn't supposed to help when redisplay cannot
keep up.  It is for the opposite use case: when redisplay is so slow
it slows down processing of your commands, like typing, for example.
You'd then set it to nil to speed up command processing, while giving
up correct display at all times.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: rudalics@gmx.at, acm@muc.de, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 5 Apr 2020 21:20:36 +0300
> 
> On 05.04.2020 21:16, Eli Zaretskii wrote:
> > No, I want fontification to work fast enough to not require these
> > measures.  They are retreat of a kind: we are giving up correctness
> > because we are annoyed by slowness.
> 
> But what's "fast enough"?

To me, "fast enough" means I don't need to stop and wait to see the
result of my typing.  Which happens now when something I type requires
reindentation and refontification, at least in some cases.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 05.04.2020 21:27, Eli Zaretskii wrote:
>>> No, I want fontification to work fast enough to not require these
>>> measures.  They are retreat of a kind: we are giving up correctness
>>> because we are annoyed by slowness.
>> But what's "fast enough"?
> To me, "fast enough" means I don't need to stop and wait to see the
> result of my typing.  Which happens now when something I type requires
> reindentation and refontification, at least in some cases.

OK, sure. But that's a different kind of scenario and benchmark.

Certainly a lower bar than having mouse-scrolling never lock up.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> apparently takes 30 to 35 calls to get into those ~10 second freezes.
[...]
> I cannot nearly simulate the problem using a simple loop that steadily
> invokes say (forward-line 50) followed by a (sit-for 0).  50 such
> invocations take 16.7 seconds here, 22.7 with (forward-line -50).

16s in one case a "10 seconds freeze": it sounds to me like you *do*
reproduce the problem.
The only difference is that with actual keyboard input pending the
redisplay is partly prevented which is what you perceive as "freeze",
but the real problem is the time it takes.

> But what's "fast enough"? Fontification that is faster than the signal rate
> of the keyboard?

Yes.  Of course, that depends not only on Emacs's code and the
optimization level, but also on your CPU and your keyboard
repetition setting.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 05.04.2020 21:23, Eli Zaretskii wrote:
> No, redisplay-dont-pause wasn't supposed to help when redisplay cannot
> keep up.  It is for the opposite use case: when redisplay is so slow
> it slows down processing of your commands, like typing, for example.
> You'd then set it to nil to speed up command processing, while giving
> up correct display at all times.

What's the difference between redisplay being slow and it being unable 
to "keep up"? I don't understand, sorry.

And in this case, IIUC, jit-lock is part of redisplay. And it's slow 
enough for this to be noticeable.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: rudalics@gmx.at, acm@muc.de, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 5 Apr 2020 21:42:48 +0300
> 
> > To me, "fast enough" means I don't need to stop and wait to see the
> > result of my typing.  Which happens now when something I type requires
> > reindentation and refontification, at least in some cases.
> 
> OK, sure. But that's a different kind of scenario and benchmark.
> 
> Certainly a lower bar than having mouse-scrolling never lock up.

I'm not sure it is lower, because as I said an optimized Emacs doesn't
lock up when I scroll, either with C-v or the mouse.  But the problems
I described do happen in such builds.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> What's the difference between redisplay being slow and it being unable to
> "keep up"? I don't understand, sorry.

If there's already input waiting (because the next event already
arrived) at the start of redisplay, then redisplay is short-circuited.

Typically, in an auto-repeat circumstance, this will either not happen
or happen on a whole batch of repetition.  So we say "can't keep up"
because processing of one command (including redisplay) takes more time
than the interval until the next command.

> And in this case, IIUC, jit-lock is part of redisplay.

Yes, it often is when scrolling.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: rudalics@gmx.at, acm@muc.de, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 5 Apr 2020 21:55:39 +0300
> 
> On 05.04.2020 21:23, Eli Zaretskii wrote:
> > No, redisplay-dont-pause wasn't supposed to help when redisplay cannot
> > keep up.  It is for the opposite use case: when redisplay is so slow
> > it slows down processing of your commands, like typing, for example.
> > You'd then set it to nil to speed up command processing, while giving
> > up correct display at all times.
> 
> What's the difference between redisplay being slow and it being unable 
> to "keep up"? I don't understand, sorry.

That's not the important difference.  The important difference is that
you want the input processed faster, without waiting for slow
redisplay.

> And in this case, IIUC, jit-lock is part of redisplay. And it's slow 
> enough for this to be noticeable.

It isn't jit-lock that's slow, it's the fontification functions it
calls.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  rudalics@gmx.at,  rrandresf@gmail.com,
>   emacs-devel@gnu.org,  rms@gnu.org,  acm@muc.de
> Date: Sun, 05 Apr 2020 15:03:14 -0400
> 
> > What's the difference between redisplay being slow and it being unable to
> > "keep up"? I don't understand, sorry.
> 
> If there's already input waiting (because the next event already
> arrived) at the start of redisplay, then redisplay is short-circuited.

Only if you call 'redisplay' from Lisp explicitly.

"Normal" redisplay will probe pending input when it is going to send
the output to the glass, and will not do so if redisplay-don't-pause
is set to nil and there's input available.  IOW, whether input
available is checked not at the start of redisplay, but somewhere in
its middle.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[andrés ramírez]

Hi. 
>> What's the difference between redisplay being slow and it being
>> unable to "keep up"? I don't understand, sorry.

Eli> That's not the important difference.  The important difference is
Eli> that you want the input processed faster, without waiting for
Eli> slow redisplay.

I remember the other day someone on this list. Having a very similar
issue as described above. Eli mentioned something about his graphic
card. But. It happens to me, when I have mesa-git with glamor
enabled. Whet I create an
emacs lucid frame and move to the scratch buffer and I write the
character 'w' (and also other identified characters but 'w' is the worst
case) that character is duplicated and also emacs lags for a few
seconds. It happens also with emacs23. On those cases I fire up qemacs or
xemacs or sxemacs those do not have the same issue.

Best Regards

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: andrés ramírez <rrandresf@gmail.com>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,
> 	rudalics@gmx.at,
> 	acm@muc.de,
> 	rms@gnu.org,
> 	emacs-devel@gnu.org
> Date: Sun, 05 Apr 2020 19:16:05 +0000
> 
> Whet I create an
> emacs lucid frame and move to the scratch buffer and I write the
> character 'w' (and also other identified characters but 'w' is the worst
> case) that character is duplicated and also emacs lags for a few
> seconds.

I suggest to run Emacs under perf and see what processing takes most
of those few seconds.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 05.04.2020 22:03, Stefan Monnier wrote:
>> What's the difference between redisplay being slow and it being unable to
>> "keep up"? I don't understand, sorry.
> If there's already input waiting (because the next event already
> arrived) at the start of redisplay, then redisplay is short-circuited.
> 
> Typically, in an auto-repeat circumstance, this will either not happen
> or happen on a whole batch of repetition.  So we say "can't keep up"
> because processing of one command (including redisplay) takes more time
> than the interval until the next command.

This agrees with my previous understanding.

>> And in this case, IIUC, jit-lock is part of redisplay.
> Yes, it often is when scrolling.

So would you agree that short-circuiting and skipping jit-lock while 
somebody leans on C-v is a good idea?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 05.04.2020 22:03, Eli Zaretskii wrote:
>>> To me, "fast enough" means I don't need to stop and wait to see the
>>> result of my typing.  Which happens now when something I type requires
>>> reindentation and refontification, at least in some cases.
>> OK, sure. But that's a different kind of scenario and benchmark.
>>
>> Certainly a lower bar than having mouse-scrolling never lock up.
> I'm not sure it is lower, because as I said an optimized Emacs doesn't
> lock up when I scroll, either with C-v or the mouse.  But the problems
> I described do happen in such builds.

It's lower because when you're typing you're usually typing something 
meaningful (meaning limited input rate), and the beginning of the window 
usually stays in place (which speeds up re-fontification).

Whereas when you're scrolling, you can scroll as fast as your keyboard 
allows you, and you're triggering fontification of a whole new 
screen-ful every keystroke. And this is closer to Martin's situation, 
AFAICT.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 05.04.2020 22:06, Eli Zaretskii wrote:

>> What's the difference between redisplay being slow and it being unable
>> to "keep up"? I don't understand, sorry.
> 
> That's not the important difference.  The important difference is that
> you want the input processed faster, without waiting for slow
> redisplay.

Two chief ways to do that are:

- To work faster
- To do less work

Ideally, we should take advantage of both. Skipping parts of redisplay 
could help with #2.

>> And in this case, IIUC, jit-lock is part of redisplay. And it's slow
>> enough for this to be noticeable.
> 
> It isn't jit-lock that's slow, it's the fontification functions it
> calls.

Yes, obviously.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Dmitry.

On Sun, Apr 05, 2020 at 21:05:00 +0300, Dmitry Gutov wrote:
> On 05.04.2020 19:15, Eli Zaretskii wrote:
> >> Do you at least see CPU activity significantly go up when you do such
> >> mouse wheel scrolling?
> > With what build?  With the -O0 build with --enable-checking, I don't
> > need the mouse: it's enough to lean on C-v and let the keyboard
> > auto-repeat do its job -- one execution unit of the CPU maxes out
> > after 5 to 10 C-v's.

> > But with a -O2 optimized production build, Emacs keeps up both when I
> > lean on C-v and when I turn the mouse wheel constantly.

> Same here. And much as I like to see certain modes improved, I'm not 
> sure this is only a performance problem. Because no matter how fast 
> syntax highlighting is, you can lean on C-v faster.

> IIRC, redisplay-dont-pause was supposed to help with things like this. 
> But it doesn't (and it's obsolete), and apparently jit-lock does its 
> thing anyway. This looks like the interesting part of the profile:

>      - scroll-up                                              5657  73%
>       - jit-lock-function                                     5444  70%
>        - jit-lock-fontify-now                                 5439  70%
>         + jit-lock--run-functions                             5431  70%
>         + run-with-timer                                         5   0%

> Do you guys want to try (setq jit-lock-defer-time 0)? It comes with a 
> certain visual downside, though.

> Or, alternatively, (setq fast-but-imprecise-scrolling t). This var seems 
> like a good idea in general, so we might consider going further with it.

Try this:

diff --git a/src/window.c b/src/window.c
index 8cdad27b66..b23ae41aba 100644
--- a/src/window.c
+++ b/src/window.c
@@ -5475,7 +5475,9 @@ window_scroll (Lisp_Object window, EMACS_INT n, bool whole, bool noerror)
 
   wset_redisplay (XWINDOW (window));
 
-  if (whole && Vfast_but_imprecise_scrolling)
+  if (Vfast_but_imprecise_scrolling
+      && (whole
+          || EQ (Vthis_command, Qmwheel_scroll)))
     specbind (Qfontification_functions, Qnil);
 
   /* On GUI frames, use the pixel-based version which is much slower
@@ -8173,6 +8175,7 @@ syms_of_window (void)
   DEFSYM (Qmode_line_format, "mode-line-format");
   DEFSYM (Qheader_line_format, "header-line-format");
   DEFSYM (Qtab_line_format, "tab-line-format");
+  DEFSYM (Qmwheel_scroll, "mwheel-scroll");
 
   DEFVAR_LISP ("temp-buffer-show-function", Vtemp_buffer_show_function,
 	       doc: /* Non-nil means call as function to display a help buffer.


This eliminates all the delay I formerly saw while mouse scrolling as
fast as possibly.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Paul Eggert]

On 4/5/20 1:46 AM, martin rudalics wrote:
>  > I looked into this, and although it's no doubt fundamentally due to a
>  > slow algorithm, the slowness is exacerbated if you use -Og (which you
>  > appear to be using).
> 
> I'm using '-O0 -g3 -no-pie' for builds that I use to debug issues and
> '-O3 -no-pie' for builds that I use for daily work.

Ah, my mistake. I thought you were using -Og (some other people were). Oh well, 
my patch does help those other people a bit.

> It offers me an overclock mode with a CPU frequency of 200 MHz and a
> PCIE Frequency of 100.  I also have M2 Boost disabled whatever that
> means.  Still no idea whether that means that I have a Black Edition.

I wouldn't overclock as it decreases reliability enough that it's not worth the 
hassle; my suggestion to overclock was meant more as a joke. Getting a newer 
computer should be a much better investment of your time/cash. If you can't 
afford one perhaps we could talk offline about the possibilities.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

Hi Alan,

On 05.04.2020 22:57, Alan Mackenzie wrote:
> Try this:
> 
> diff --git a/src/window.c b/src/window.c
> index 8cdad27b66..b23ae41aba 100644
> --- a/src/window.c
> +++ b/src/window.c
> @@ -5475,7 +5475,9 @@ window_scroll (Lisp_Object window, EMACS_INT n, bool whole, bool noerror)
>   
>     wset_redisplay (XWINDOW (window));
>   
> -  if (whole && Vfast_but_imprecise_scrolling)
> +  if (Vfast_but_imprecise_scrolling
> +      && (whole
> +          || EQ (Vthis_command, Qmwheel_scroll)))
>       specbind (Qfontification_functions, Qnil);
>   
>     /* On GUI frames, use the pixel-based version which is much slower
> @@ -8173,6 +8175,7 @@ syms_of_window (void)
>     DEFSYM (Qmode_line_format, "mode-line-format");
>     DEFSYM (Qheader_line_format, "header-line-format");
>     DEFSYM (Qtab_line_format, "tab-line-format");
> +  DEFSYM (Qmwheel_scroll, "mwheel-scroll");

You probably meant to address Martin, but I can just say that while this 
will almost certainly work, it doesn't address the scenario of flooring 
C-v. Or M-v. Etc. Which seem equivalent.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>>> And in this case, IIUC, jit-lock is part of redisplay.
>> Yes, it often is when scrolling.
> So would you agree that short-circuiting and skipping jit-lock while
> somebody leans on C-v is a good idea?

It sounds like a trick question, because what I just described says that
it is what already happens, more or less, and I'm not sure which part of
the difference you're thinking of.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: rudalics@gmx.at, acm@muc.de, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 5 Apr 2020 22:25:16 +0300
> 
> >> Certainly a lower bar than having mouse-scrolling never lock up.
> > I'm not sure it is lower, because as I said an optimized Emacs doesn't
> > lock up when I scroll, either with C-v or the mouse.  But the problems
> > I described do happen in such builds.
> 
> It's lower because when you're typing you're usually typing something 
> meaningful (meaning limited input rate), and the beginning of the window 
> usually stays in place (which speeds up re-fontification).
> 
> Whereas when you're scrolling, you can scroll as fast as your keyboard 
> allows you, and you're triggering fontification of a whole new 
> screen-ful every keystroke.

And yet Emacs does cope with that "higher bar", whereas while I type I
sometimes see unbearably slow responses.  Which in my book doesn't fit
its being "lower bar".

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Date: Sun, 5 Apr 2020 19:57:53 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: martin rudalics <rudalics@gmx.at>, Eli Zaretskii <eliz@gnu.org>,
>  rrandresf@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> 
> Try this:
> 
> diff --git a/src/window.c b/src/window.c
> index 8cdad27b66..b23ae41aba 100644
> --- a/src/window.c
> +++ b/src/window.c
> @@ -5475,7 +5475,9 @@ window_scroll (Lisp_Object window, EMACS_INT n, bool whole, bool noerror)
>  
>    wset_redisplay (XWINDOW (window));
>  
> -  if (whole && Vfast_but_imprecise_scrolling)
> +  if (Vfast_but_imprecise_scrolling
> +      && (whole
> +          || EQ (Vthis_command, Qmwheel_scroll)))
>      specbind (Qfontification_functions, Qnil);

That's just to make a point, no?  You don't seriously propose such a
change, right?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Recent Emacsen either ignore that variable or silently reset it to nil
  > internally so it doesn't get into their way.  Their progmodes either
  > always scan an entire buffer from its beginning or use some elaborate,
  > fragile techniques to find such a top level position.  Moreover, our
  > underlying mechanism for syntax highlighting always marks the entire
  > rest of a buffer as dirty after every single editing change.  This has
  > the consequence that that entire part has to be continuously rescanned
  > when some of it is shown in another window.

Does anyone disagree with this specific factual claim?

  > The basic slowness of Emacs over the past years is a direct consequence
  > of that policy.

Does anyone disagree with this general claim?

  > So since you earlier asked for "a switch to turn off the changes" then
  > my answer is that such a switch already exists but has been deactivated.

If those claims are true, I suggest reactivating it.
Why impose the fast=machine mode on everyone?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >   > This assumption proved to be very problematic.

  > > Problematic for whom?

  > Well, for me for a start.

If it causes a problem only for you personally as a maintainer, I am
sure we could find a way to spare you from spending time on this.  But
we are miscommunicating completely.

I assumed you meant "problematically for some people who are doing
some kinds of things with Emacs", and I would like to know what kinds
of things they are.  The answer that is crucial for deciding whether
this problem ought to have been "fixed" or not.

  > Suppose open-paren-in-column-0-is-defun-start is non-nil, and we have
  > something like:

  >     {
  >     /* foo
  >     (but bar
  >     */
  >      }
  >     ^
  >   point

Now I see the scenario.  Indeed, with
open-paren-in-column-0-is-defun-start non-nil, some users would regard
the resulting behavior as a problem.

I do not see how this constitutes a reason to deliminate that option.
Why is that better than the obvious response: to tell the user,

   Either delate that open-paren, or indent it, or set
   open-paren-in-column-0-is-defun-start to nil.

Does someone want to present an argument leading to the conclusion
that it was better to deactivate that variable?

  > I think I meant that, given their validity, it is up to us as Emacs
  > developers to arrange that they don't cause trouble, rather than
  > expecting our users to insert these obtrusive backslashes.

Why assume that the one recommended way to edit the buffer to avoid
the confusion is inserting a backslash?


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >   > This assumption proved to be very problematic.

  > > Problematic for whom?

  > Well, for me for a start.

If it causes a problem only for you personally as a maintainer, I am
sure we could find a way to spare you from spending time on this.  But
I think we are miscommunicating over this.

I understood you to mean "problematic for some people who are doing
some kinds of things with Emacs", so I would like to know what kinds
of things they are.  The answer that is crucial for deciding whether
this problem ought to have been "fixed" or not.

But is that what you meant?  Or did you mean something else?

I get the impression that your analysis of the issue is based on
assuming that open-paren-in-column-0-is-defun-start defaults to t.
What if it defaulted to nil?  Would that eliminate the problem
you're concerned about?  I think so.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Indeed.  And the reverse is true as well: while it might be OK for
  > a user to put his open-parens in column 0 to help Emacs be more
  > responsive, it's not OK for Emacs to be too slow on files where there
  > are no such open-parens in column 0.  So Emacs has to use some other
  > mechanism anyway to avoid the slowdown in such files.

The mechanism it has now is effective only on fast machines.

							   Once you have
  > this mechanism in place, the "open-parens in column 0" heuristic is just
  > not worth the trouble any more.

If and when such a mechanism is in place for us free software
activists, we might agree that open-paren-in-column-0-is-defun-start
is not worth while any more.  But it appears that has not happened
yet, so I think that the open-paren-in-column-0-is-defun-start guided
heuristic IS still worth while for us.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Ravine Var]

martin rudalics <rudalics@gmx.at> writes:
> I meanwhile suspect that one culprit might be the progressive squaring
> in 'mwheel-scroll'.  But I don't understand that yet and do not see it
> documented well.  Also, setting 'mouse-wheel-scroll-amount' to '(1) does
> not seem to make any difference either.

I have set mouse-wheel-progressive-speed to nil to get somewhat
reasonable scrolling behavior. Perhaps this tweak can make a difference
in your machine...

-R

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > I appreciate you needing to run in a debug build,

I run a debug build only only when I try to investigate an error
reported on bug-gnu-emacs or when I have to run Emacs under GDB.  Maybe
optimized builds can deliver acceptable behavior now when run under GDB.
But the last time I tried, behavior was not acceptable.  And what about
our advice that users should run Emacs under GDB all the time to detect
the cause of a crash?

 > but what sort of
 > timing do you get on an optimised build?

An optimized build runs reasonably fast here so I didn't bother so far.

 > I think you said you have a state of the art 2007 processor.  Mine is
 > from 2017.  Processing power did not speed up by a factor of 100 in that
 > period.

That's why I wondered that nobody else sees this problem.  Does really
nobody run a debug build nowadays?

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > I also can't imagine how defaulting to nil "would have been completely
 > sufficient IMHO": setting the var to nil had been an option for decades
 > but it never enjoyed much popularity.

Defaulting to nil would have meant that people who want "correct" syntax
under all conditions would never have to care about this.  OTOH, people
who did care about performance more would have had the possibility to
set this option to non-nil and get the necessary feedback via
font-locking (highlighting open parens in column zero red when they are
in a string or a comment) when they accidentally inserted such a paren
or scrolled into it.  Why was that highlighting functionality removed?

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >> I cannot nearly simulate the problem using a simple loop that steadily
 >> invokes say (forward-line 50) followed by a (sit-for 0).  50 such
 >> invocations take 16.7 seconds here, 22.7 with (forward-line -50).  And
 >> redisplay takes place as expected here.  So it's likely the mouse wheel
 >> input mounting up that inhibits redisplay.
 >
 > Not forward-line, scroll-up.  That's what wheel.el calls.

I know.  But I wanted to not relate the behavior to things like the
window configuration.

 >> Do you at least see CPU activity significantly go up when you do such
 >> mouse wheel scrolling?
 >
 > With what build?  With the -O0 build with --enable-checking, I don't
 > need the mouse: it's enough to lean on C-v and let the keyboard
 > auto-repeat do its job -- one execution unit of the CPU maxes out
 > after 5 to 10 C-v's.

OK.  Then people see the problem and it's just that they do not build
with -O0.

 > But with a -O2 optimized production build, Emacs keeps up both when I
 > lean on C-v and when I turn the mouse wheel constantly.

Same here.

 > Anyway, you don't need to work too hard to get me started on how slow
 > CC Mode is, especially when re-indentation is involved in addition to
 > font-lock.  Which is why I think we should move away of regexp-based
 > and SMIE-based engines towards fast parsers written in C.

As long as we are not there (and IMO even after that) Emacs should be
able to do its SMIE parsing in a practical way: Restrict backward and
forward parsing to the smallest reasonable code fragment around point.
And reasonable would mean the smallest enclosing fragment delimited by
two parens in column zero it can find in either direction (which can be
still quite large when viewing functions like redisplay_internal).

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >> I cannot nearly simulate the problem using a simple loop that steadily
 >> invokes say (forward-line 50) followed by a (sit-for 0).  50 such
 >> invocations take 16.7 seconds here, 22.7 with (forward-line -50).
 >
 > 16s in one case a "10 seconds freeze": it sounds to me like you *do*
 > reproduce the problem.

Not really.  In the 16s case redisplay kicks in and displays the buffer
250 times.  In the 10s case redisplay stops after five times.

 > The only difference is that with actual keyboard input pending the
 > redisplay is partly prevented which is what you perceive as "freeze",
 > but the real problem is the time it takes.

What does get fontified when redisplay freezes?

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > I have set mouse-wheel-progressive-speed to nil to get somewhat
 > reasonable scrolling behavior. Perhaps this tweak can make a difference
 > in your machine...

It doesn't help.  I already tried setting ‘mouse-wheel-scroll-amount’ to
'(1) which should do about the same.  The interference of these two
options is not yet clear to me.

Thanks, martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Richard.

On Sun, Apr 05, 2020 at 22:36:22 -0400, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

>   > >   > This assumption proved to be very problematic.

>   > > Problematic for whom?

>   > Well, for me for a start.

> If it causes a problem only for you personally as a maintainer, I am
> sure we could find a way to spare you from spending time on this.  But
> we are miscommunicating completely.

> I assumed you meant "problematically for some people who are doing
> some kinds of things with Emacs", and I would like to know what kinds
> of things they are.  The answer that is crucial for deciding whether
> this problem ought to have been "fixed" or not.

Yes, that is what I meant.  The "kind of things" is nothing more than
writing comments in C (etc.) source code.  Sooner or later, such a
comment has a parenthesis in it, and sooner or later (e.g. by M-q), this
will end up in column 0, when (as is often the case) the user has no
" *" comment prefix.

>   > Suppose open-paren-in-column-0-is-defun-start is non-nil, and we have
>   > something like:

>   >     {
>   >     /* foo
>   >     (but bar
>   >     */
>   >      }
>   >     ^
>   >   point

> Now I see the scenario.  Indeed, with
> open-paren-in-column-0-is-defun-start non-nil, some users would regard
> the resulting behavior as a problem.

[ Please do M-: (setq comment-use-syntax-ppss nil) to reenable
o-p-i-c-0-i-d-s's action. ]

I think "some" is an understatement.  With show-paren-mode enabled, and
with point after the }, it visibly mismatches the ( rather than matching
the {.

If this was all that happened, it might not be too bad.  But
(forward-comment -1) fails on such a beast.  scan-lists (backwards)
fails, too.  We're talking about low level Emacs primitives being buggy.
This is surely worse than higher level functions being buggy.

> I do not see how this constitutes a reason to deliminate that option.
> Why is that better than the obvious response: to tell the user,

>    Either delate that open-paren, or indent it, or set
>    open-paren-in-column-0-is-defun-start to nil.

Because it is highly non-trivial to detect this happening in the code.
To insert code to detect this would cause a slow down, likely a big slow
down.

> Does someone want to present an argument leading to the conclusion
> that it was better to deactivate that variable?

This has, I think, been much discussed on this list in years past.  I
actually implemented a solution whereby comments and strings would have
been marked by a text property in forward scanning, and back_comment
would merely examine that property.  This scheme wasn't accepted for
Emacs, and it would have suffered the same from needing large scale
scanning.

My position at the moment is that the o-p-i-c-0-i-d-s mechanism should
remain active, but that we should fix the bugs in it such that the
backward scanning would only stop at a column-0 paren not in a comment
or string.  I outlined one way of doing this in the thread called "A
proposal for the future of open-paren-in-column-0-is-defun-start", but
nobody has responded to the thrust of this post at all, as yet.

>   > I think I meant that, given their validity, it is up to us as Emacs
>   > developers to arrange that they don't cause trouble, rather than
>   > expecting our users to insert these obtrusive backslashes.

> Why assume that the one recommended way to edit the buffer to avoid
> the confusion is inserting a backslash?

This was the recommendation which used to be in the Emacs manual.  (I
think I took it out.)  When bug #22884 hit, Paul Eggert "solved" it by
refilling the copyright text such that the paren wasn't in column 0
anymore.

> -- 
> Dr Richard Stallman
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Eli.

On Mon, Apr 06, 2020 at 05:27:22 +0300, Eli Zaretskii wrote:
> > Date: Sun, 5 Apr 2020 19:57:53 +0000
> > From: Alan Mackenzie <acm@muc.de>
> > Cc: martin rudalics <rudalics@gmx.at>, Eli Zaretskii <eliz@gnu.org>,
> >  rrandresf@gmail.com, rms@gnu.org, emacs-devel@gnu.org

> > Try this:

> > diff --git a/src/window.c b/src/window.c
> > index 8cdad27b66..b23ae41aba 100644
> > --- a/src/window.c
> > +++ b/src/window.c
> > @@ -5475,7 +5475,9 @@ window_scroll (Lisp_Object window, EMACS_INT n, bool whole, bool noerror)

> >    wset_redisplay (XWINDOW (window));

> > -  if (whole && Vfast_but_imprecise_scrolling)
> > +  if (Vfast_but_imprecise_scrolling
> > +      && (whole
> > +          || EQ (Vthis_command, Qmwheel_scroll)))
> >      specbind (Qfontification_functions, Qnil);

> That's just to make a point, no?  You don't seriously propose such a
> change, right?

Well, the point is that extending the domain of
fast-but-imprecise-scrolling isn't difficult.

As to whether or not it's a good idea, probably not in such a crude
manner.  I think I would favour such an extension, yes, but I don't feel
strongly about it.  It would solve the problem of the 10 second wait at
the end of crazed mouse wheel scrolling, though.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 05.04.2020 23:48, Dmitry Gutov wrote:
> 
> You probably meant to address Martin, but I can just say that while this 
> will almost certainly work, it doesn't address the scenario of flooring 
> C-v. Or M-v. Etc. Which seem equivalent.

Apologies. I read the patch wrong. Apparently before this patch the 
optimization didn't [always] apply to mouse scrolling?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: acm@muc.de, eliz@gnu.org, rrandresf@gmail.com, emacs-devel@gnu.org
> Date: Sun, 05 Apr 2020 22:36:20 -0400
> 
>   > Recent Emacsen either ignore that variable or silently reset it to nil
>   > internally so it doesn't get into their way.  Their progmodes either
>   > always scan an entire buffer from its beginning or use some elaborate,
>   > fragile techniques to find such a top level position.  Moreover, our
>   > underlying mechanism for syntax highlighting always marks the entire
>   > rest of a buffer as dirty after every single editing change.  This has
>   > the consequence that that entire part has to be continuously rescanned
>   > when some of it is shown in another window.
> 
> Does anyone disagree with this specific factual claim?

I'm not sure what is "the claim" here, but I want to point out a small
inaccuracy: redisplay doesn't "continuously rescan the entire rest of
the buffer", it only rescans the part(s) shown in windows.  (It might
happen that some major mode's font-lock definitions end up rescanning
much more, but that's a separate issue.)  And frankly, what would we
like Emacs to do instead?  A change in a buffer can potentially affect
the fontification of the rest of the buffer, and I don't think we
would like Emacs to fail to update other windows showing the same
buffer -- that would be against Emacs's description as "real-time"
editor.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rrandresf@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Mon, 6 Apr 2020 11:05:11 +0200
> 
>  >> Do you at least see CPU activity significantly go up when you do such
>  >> mouse wheel scrolling?
>  >
>  > With what build?  With the -O0 build with --enable-checking, I don't
>  > need the mouse: it's enough to lean on C-v and let the keyboard
>  > auto-repeat do its job -- one execution unit of the CPU maxes out
>  > after 5 to 10 C-v's.
> 
> OK.  Then people see the problem and it's just that they do not build
> with -O0.

I build with -O0 all the time, it's just that, like you, I never use
such a build for any serious work, only for debugging and exploring of
new features.

> As long as we are not there (and IMO even after that) Emacs should be
> able to do its SMIE parsing in a practical way: Restrict backward and
> forward parsing to the smallest reasonable code fragment around point.
> And reasonable would mean the smallest enclosing fragment delimited by
> two parens in column zero it can find in either direction (which can be
> still quite large when viewing functions like redisplay_internal).

That's for the major modes to implement, the jit-lock infrastructure
calls the fontification functions on a small region around the area of
interest of the display engine.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Cc: martin rudalics <rudalics@gmx.at>, Eli Zaretskii <eliz@gnu.org>,
>  rrandresf@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> Date: Mon, 6 Apr 2020 15:43:17 +0300
> 
> On 05.04.2020 23:48, Dmitry Gutov wrote:
> > 
> > You probably meant to address Martin, but I can just say that while this 
> > will almost certainly work, it doesn't address the scenario of flooring 
> > C-v. Or M-v. Etc. Which seem equivalent.
> 
> Apologies. I read the patch wrong. Apparently before this patch the 
> optimization didn't [always] apply to mouse scrolling?

It affects any scroll command, but only partially.

People often forget that scrolling is implemented partially in the
command that is invoked by the likes of C-v, and partially in the
display engine.  Binding a variable in the command only affects the
former part, but not the latter.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 06.04.2020 16:39, Eli Zaretskii wrote:
> People often forget that scrolling is implemented partially in the
> command that is invoked by the likes of C-v, and partially in the
> display engine.  Binding a variable in the command only affects the
> former part, but not the latter.

I thought binding fontification-functions to nil was done there exactly 
with that intent: to affect what the display engine subsequently does. 
Isn't that right?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>   > Indeed.  And the reverse is true as well: while it might be OK for
>   > a user to put his open-parens in column 0 to help Emacs be more
>   > responsive, it's not OK for Emacs to be too slow on files where there
>   > are no such open-parens in column 0.  So Emacs has to use some other
>   > mechanism anyway to avoid the slowdown in such files.
>
> The mechanism it has now is effective only on fast machines.

Do you have any evidence to back that claim?


        Stefan "whose main work machines are Core 2 Duo"

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 6 Apr 2020 16:55:11 +0300
> 
> On 06.04.2020 16:39, Eli Zaretskii wrote:
> > People often forget that scrolling is implemented partially in the
> > command that is invoked by the likes of C-v, and partially in the
> > display engine.  Binding a variable in the command only affects the
> > former part, but not the latter.
> 
> I thought binding fontification-functions to nil was done there exactly 
> with that intent: to affect what the display engine subsequently does. 

Can't be done, because redisplay runs only _after_ the scroll command
returns, at which point this binding is undone.  You can easily see
this in a debugger: put a breakpoint inside redisplay_internal right
after the specbind line is executed, then type

  (gdb) p Fsymbol_value (Qfontification_functions)

You will see the value is a cons cell, not nil.

What that binding does is affect the portions of display code that are
run as part of window_scroll_pixel_based, which the scroll command
calls to calculate a suitable position of window-start by simulating
display.  That display code runs with fontifications disabled, which
makes it faster.

All the scroll commands in Emacs do is more or less calculate the
window-start point; the rest is done when redisplay kicks in.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>   > Recent Emacsen either ignore that variable or silently reset it to nil
>   > internally so it doesn't get into their way.  Their progmodes either
>   > always scan an entire buffer from its beginning or use some elaborate,
>   > fragile techniques to find such a top level position.  Moreover, our
>   > underlying mechanism for syntax highlighting always marks the entire
>   > rest of a buffer as dirty after every single editing change.  This has
>   > the consequence that that entire part has to be continuously rescanned
>   > when some of it is shown in another window.
>
> Does anyone disagree with this specific factual claim?

There are several claims above which can be interpreted in many
different ways, some of them correspond to reality but many others
don't.  So if you don't know the actual facts behind the above claims
you're likely to infer incorrect conclusions.

>   > The basic slowness of Emacs over the past years is a direct consequence
>   > of that policy.
> Does anyone disagree with this general claim?

I think I made it clear that I do.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 06.04.2020 05:36, Richard Stallman wrote:
> If and when such a mechanism is in place for us free software
> activists, we might agree that open-paren-in-column-0-is-defun-start
> is not worth while any more.  But it appears that has not happened
> yet, so I think that the open-paren-in-column-0-is-defun-start guided
> heuristic IS still worth while for us.

This question (whether syntax-ppss is "too slow") was raised a couple of 
times already, but in all previous discussions, as well as in this one, 
nobody has done the bare minimum research (e.g. profiling some 
real-world scenario) to prove this claim. Nor, if it indeed showed up in 
profiler reports, examined which part of the current mechanism is the 
bottleneck. Nor tried to fix that part somehow.

This is not how we should approach performance problems.

The only arguments I see are nostalgic (e.g., somehow, Emacs was faster 
years ago, maybe? the files were smaller, and the C standard was simpler?)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>> 16s in one case a "10 seconds freeze": it sounds to me like you *do*
>> reproduce the problem.
> Not really.  In the 16s case redisplay kicks in and displays the buffer
> 250 times.  In the 10s case redisplay stops after five times.

Yes, but the underlying problem remains: it takes 10-16s to do the work,
so if the corresponding input events come in over a period shorter than
10-16s Emacs won't be able to keep up.

>> The only difference is that with actual keyboard input pending the
>> redisplay is partly prevented which is what you perceive as "freeze",
>> but the real problem is the time it takes.
> What does get fontified when redisplay freezes?

The text gets fontified by the scrolling routine because the size of the
text depends on the fontification, so correct scrolling needs to fontify
the text before it can compute the destination of the scroll.

That's where `fast-but-imprecise-scrolling` comes into play.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> As long as we are not there (and IMO even after that) Emacs should be
> able to do its SMIE parsing in a practical way: Restrict backward and
> forward parsing to the smallest reasonable code fragment around point.
> And reasonable would mean the smallest enclosing fragment delimited by
> two parens in column zero it can find in either direction (which can be
> still quite large when viewing functions like redisplay_internal).

Could people refrain from "X should do Y" when they have no idea how
X currently works?

SMIE's indentation works by parsing backward, so that it stops parsing
as soon as it found the info needed to determine the indentation to use.

It almost always stops long before reaching the nearest "open paren in
column 0".


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: acm@muc.de,  Eli Zaretskii <eliz@gnu.org>,  rrandresf@gmail.com,
>   rms@gnu.org,  emacs-devel@gnu.org
> Date: Mon, 06 Apr 2020 11:25:43 -0400
> 
> >> 16s in one case a "10 seconds freeze": it sounds to me like you *do*
> >> reproduce the problem.
> > Not really.  In the 16s case redisplay kicks in and displays the buffer
> > 250 times.  In the 10s case redisplay stops after five times.
> 
> Yes, but the underlying problem remains: it takes 10-16s to do the work,
> so if the corresponding input events come in over a period shorter than
> 10-16s Emacs won't be able to keep up.

I think "the work" of a single scroll-up takes less (much less) than
10-16s.  But when several C-v keypresses accumulate (because the user
types them faster than Emacs can process them), their collective work
takes a long time to process.

> The text gets fontified by the scrolling routine because the size of the
> text depends on the fontification, so correct scrolling needs to fontify
> the text before it can compute the destination of the scroll.
> 
> That's where `fast-but-imprecise-scrolling` comes into play.

Right.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>> >> 16s in one case a "10 seconds freeze": it sounds to me like you *do*
>> >> reproduce the problem.
>> > Not really.  In the 16s case redisplay kicks in and displays the buffer
>> > 250 times.  In the 10s case redisplay stops after five times.
>> 
>> Yes, but the underlying problem remains: it takes 10-16s to do the work,
>> so if the corresponding input events come in over a period shorter than
>> 10-16s Emacs won't be able to keep up.
>
> I think "the work" of a single scroll-up takes less (much less) than
> 10-16s.

Indeed, IIUC his measurement of 10-16s is for the equivalent of about
50 scrolls.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 06.04.2020 17:44, Eli Zaretskii wrote:

>> I thought binding fontification-functions to nil was done there exactly
>> with that intent: to affect what the display engine subsequently does.
> 
> Can't be done, because redisplay runs only _after_ the scroll command
> returns, at which point this binding is undone.  You can easily see
> this in a debugger: put a breakpoint inside redisplay_internal right
> after the specbind line is executed, then type
> 
>    (gdb) p Fsymbol_value (Qfontification_functions)
> 
> You will see the value is a cons cell, not nil.
> 
> What that binding does is affect the portions of display code that are
> run as part of window_scroll_pixel_based, which the scroll command
> calls to calculate a suitable position of window-start by simulating
> display.  That display code runs with fontifications disabled, which
> makes it faster.
> 
> All the scroll commands in Emacs do is more or less calculate the
> window-start point; the rest is done when redisplay kicks in.

So what happens then? Does it trigger jit-lock anyway?

If so, I don't understand how we get the apparent performance 
improvement from using fast-but-imprecise-scrolling.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >> As long as we are not there (and IMO even after that) Emacs should be
 >> able to do its SMIE parsing in a practical way: Restrict backward and
 >> forward parsing to the smallest reasonable code fragment around point.
 >> And reasonable would mean the smallest enclosing fragment delimited by
 >> two parens in column zero it can find in either direction (which can be
 >> still quite large when viewing functions like redisplay_internal).
 >
 > Could people refrain from "X should do Y" when they have no idea how
 > X currently works?
 >
 > SMIE's indentation works by parsing backward, so that it stops parsing
 > as soon as it found the info needed to determine the indentation to use.
 >
 > It almost always stops long before reaching the nearest "open paren in
 > column 0".

I did interpret Eli's remark about "SMIE-based engines" as the simple
minded approach used by CC-mode which to my knowledge often does not
stop long before reaching the nearest "open paren in column 0".  I did
not intend to relate this to smie.el but it's obvious that I created
that impression.  Sorry for that.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 06.04.2020 05:16, Stefan Monnier wrote:
>>>> And in this case, IIUC, jit-lock is part of redisplay.
>>> Yes, it often is when scrolling.
>> So would you agree that short-circuiting and skipping jit-lock while
>> somebody leans on C-v is a good idea?
> It sounds like a trick question, because what I just described says that
> it is what already happens, more or less, and I'm not sure which part of
> the difference you're thinking of.

We do that when fast-but-imprecise-scrolling is on, right?

But with it off (which is the default), fontification functions are 
still forced to run during every scroll-up command, don't they?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 06.04.2020 05:25, Eli Zaretskii wrote:
>>>> Certainly a lower bar than having mouse-scrolling never lock up.
>>> I'm not sure it is lower, because as I said an optimized Emacs doesn't
>>> lock up when I scroll, either with C-v or the mouse.  But the problems
>>> I described do happen in such builds.
>> It's lower because when you're typing you're usually typing something
>> meaningful (meaning limited input rate), and the beginning of the window
>> usually stays in place (which speeds up re-fontification).
>>
>> Whereas when you're scrolling, you can scroll as fast as your keyboard
>> allows you, and you're triggering fontification of a whole new
>> screen-ful every keystroke.
> And yet Emacs does cope with that "higher bar", whereas while I type I
> sometimes see unbearably slow responses.  Which in my book doesn't fit
> its being "lower bar".

At least it's lower in theory. But certain implementation approaches can 
make it more difficult.

Would I be correct to assume that you see only see this in CC Mode based 
major mode, and only when editing big enough files? Such as xdisp.c 
mentioned many times recently.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Dmitry.

On Mon, Apr 06, 2020 at 17:55:50 +0300, Dmitry Gutov wrote:
> On 06.04.2020 05:36, Richard Stallman wrote:
> > If and when such a mechanism is in place for us free software
> > activists, we might agree that open-paren-in-column-0-is-defun-start
> > is not worth while any more.  But it appears that has not happened
> > yet, so I think that the open-paren-in-column-0-is-defun-start guided
> > heuristic IS still worth while for us.

> This question (whether syntax-ppss is "too slow") was raised a couple of 
> times already, but in all previous discussions, as well as in this one, 
> nobody has done the bare minimum research (e.g. profiling some 
> real-world scenario) to prove this claim. Nor, if it indeed showed up in 
> profiler reports, examined which part of the current mechanism is the 
> bottleneck. Nor tried to fix that part somehow.

> This is not how we should approach performance problems.

> The only arguments I see are nostalgic (e.g., somehow, Emacs was faster 
> years ago, maybe? the files were smaller, and the C standard was simpler?)

I've attempted to simulate one of Martin's scenarios, and set up my tty
frame with two side by side windows, each 65 lines deep and wide enough
for xdisp.c not to need continuation lines.  xdisp.c is displayed in both
windows, from Line 401 at the LHS, and from Line 35162 on the RHS.  In
the LH window point is at BOL 435 (the opening brace for function
fill_column_indicator_column) and in the RH window point is at BOL 35159
(a random line in start_hourglass).

With various combinations of the pertinent variables, I run

    (time-it (insert "{") (other-window 1) (insert " ") (other-window 1)
    (sit-for 0))

.  After each such timing I undo the buffer changes before the next
timing.

Here are the results:
                         open-paren-in-column-zero-is-defun-start
			      nil                    t
comment-use-syntax-ppss
        nil                  0.319s                0.264s
	 t                   0.319s                0.227s

.  Bearing in mind that c-u-s-p being t suppresses the action of
o-p-i-c-0-i-d-s in back_comment, but not in beginning-of-defun, it seems
like the o-p-i-c-0-i-d-s mechanism does indeed speed up some scenarios
in C Mode, significantly but not massively.

IMAO, It would be nice to have the code testing o-p-i-c-0-i-d-s (both
places) able to ignore spurious cases of parens in literals.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>>>>> And in this case, IIUC, jit-lock is part of redisplay.
>>>> Yes, it often is when scrolling.
>>> So would you agree that short-circuiting and skipping jit-lock while
>>> somebody leans on C-v is a good idea?
>> It sounds like a trick question, because what I just described says that
                                                 ^
                                                you (or "is")

>> it is what already happens, more or less, and I'm not sure which part of
>> the difference you're thinking of.
> We do that when fast-but-imprecise-scrolling is on, right?

At this point I don't really know "that" refers to.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 07.04.2020 00:27, Stefan Monnier wrote:
>>>>>> And in this case, IIUC, jit-lock is part of redisplay.
>>>>> Yes, it often is when scrolling.
>>>> So would you agree that short-circuiting and skipping jit-lock while
>>>> somebody leans on C-v is a good idea?
>>> It sounds like a trick question, because what I just described says that
>                                                   ^
>                                                  you (or "is")
> 
>>> it is what already happens, more or less, and I'm not sure which part of
>>> the difference you're thinking of.
>> We do that when fast-but-imprecise-scrolling is on, right?
> At this point I don't really know "that" refers to.

that = skip jit-lock when input is pending.

If not, where else its performance improvement comes from?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

Hi Alan,

On 06.04.2020 22:36, Alan Mackenzie wrote:

> I've attempted to simulate one of Martin's scenarios, and set up my tty
> frame with two side by side windows, each 65 lines deep and wide enough
> for xdisp.c not to need continuation lines.  xdisp.c is displayed in both
> windows, from Line 401 at the LHS, and from Line 35162 on the RHS.  In
> the LH window point is at BOL 435 (the opening brace for function
> fill_column_indicator_column) and in the RH window point is at BOL 35159
> (a random line in start_hourglass).
> 
> With various combinations of the pertinent variables, I run
> 
>      (time-it (insert "{") (other-window 1) (insert " ") (other-window 1)
>      (sit-for 0))
> 
> .  After each such timing I undo the buffer changes before the next
> timing.

So you chose the absolute worst scenario for syntax-ppss, right? Where 
its cache would be unavoidably clobbered at least once. I'm guessing the 
difference between measurements is ~ how long (parse-partial-sexp 
(point-min) (point-max)) takes on your machine in that file.

How about some other, more common scenarios?

Like Martin just scrolling the buffer (we can simulate that with several 
calls to scroll-up).

Or Eli just typing in some large file.

I think these two cases featured more prominently in the recent 
discussions, with associated complaints about performance.

> Here are the results:
>                           open-paren-in-column-zero-is-defun-start
> 			      nil                    t
> comment-use-syntax-ppss
>          nil                  0.319s                0.264s
> 	 t                   0.319s                0.227s
> 
> .  Bearing in mind that c-u-s-p being t suppresses the action of
> o-p-i-c-0-i-d-s in back_comment, but not in beginning-of-defun, it seems
> like the o-p-i-c-0-i-d-s mechanism does indeed speed up some scenarios
> in C Mode, significantly but not massively.
> 
> IMAO, It would be nice to have the code testing o-p-i-c-0-i-d-s (both
> places) able to ignore spurious cases of parens in literals.

I can't imagine anybody doing the 
type-switch-window-to-the-same-buffer-type routine frequently, but if 
you really want to improve that use case, I think the first takeaway 
that the necessary knobs are still present. A text property check 
wouldn't slow the routines significantly (I think), but you'd have to 
benchmark that as well, and also the extra code in CC Mode which would 
scan the whole buffer and apply those properties. In the first 
implementation, at least, you could do that using the normal 
before/after-change-functions, not "super" ones.

Here's a better idea to try, though: keep comment-use-syntax-ppss set to 
t. And also use syntax-ppss to find the beginning of defun in 
c-beginning-of-defun. But to make it honor the o-p-i-c-0-i-d-s, set 
syntax-begin-function to a new function that contains whatever "fast" 
logic you prefer. E.g. scanning the buffer back for open parens in 
column 1 without certain text property applied. I'm not 100% sure why 
this var is marked obsolete now (probably because it creates murky 
semantics), but this is the simplest way to add this behavior, I think.

In any case, I'm afraid all this won't bring any dramatic improvement in 
the more usual scenarios which I mentioned above. And we'll be back to 
looking for other ways to improve performance.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> So you chose the absolute worst scenario for syntax-ppss, right? Where its
> cache would be unavoidably clobbered at least once. I'm guessing the
> difference between measurements is ~ how long (parse-partial-sexp 
> (point-min) (point-max)) takes on your machine in that file.

FWIW, `syntax-ppss` has not been optimized for this case at all, so
I wouldn't be surprised if it takes noticeably more than

    (parse-partial-sexp (point-min) (point-max))

because it performs that computation in chunks, and the overhead for
each chunk is likely non-negligible.
[ Note: this is just a wild guess on my part.  I never bothered to
  measure it: `syntax-ppss` is a fairly naive implementation with a lot
  of room for improvement.  ]

IOW if it takes the same time as (parse-partial-sexp (point-min) (point-max))
then I'm quite happy ;-)

> How about some other, more common scenarios?

I agree that it's probably not very representative.

>> Here are the results:
>>                           open-paren-in-column-zero-is-defun-start
>> 			      nil                    t
>> comment-use-syntax-ppss
>>          nil                  0.319s                0.264s
>> 	 t                   0.319s                0.227s
>> .  Bearing in mind that c-u-s-p being t suppresses the action of
>> o-p-i-c-0-i-d-s in back_comment, but not in beginning-of-defun, it seems
>> like the o-p-i-c-0-i-d-s mechanism does indeed speed up some scenarios
>> in C Mode, significantly but not massively.
>> IMAO, It would be nice to have the code testing o-p-i-c-0-i-d-s (both
>> places) able to ignore spurious cases of parens in literals.

The problem here is that in order to decide whether or not it's *still*
spurious after the change near BOB, you basically have to compute the
equivalent of `syntax-ppss`.  So we're back to square one.

What could be done is to change `syntax-ppss` so it optimizes for the
case where the buffer changes do not impact the way parsing is done
"further down" (i.e. not for the case where the change just opened (or
closed) a string/comment).  Not sure it's worth the trouble, OTOH.

> 1 without certain text property applied. I'm not 100% sure why this var is
> marked obsolete now (probably because it creates murky semantics), but this
> is the simplest way to add this behavior, I think.

Yes, the problem is the murky semantics (including the fact that you
can't rely on it being used, even though some code does want to rely on
it, typically in multi-mode setups where it's abused to impose
a boundary).

> In any case, I'm afraid all this won't bring any dramatic improvement in the
> more usual scenarios which I mentioned above. And we'll be back to looking
> for other ways to improve performance.

That's also my impression.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> that = skip jit-lock when input is pending.

AFAIK jit-lock is never skipped: it's (part of) the redisplay which is
skipped (and that in turn may include calls to jit-lock).


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 07.04.2020 02:43, Stefan Monnier wrote:
>> that = skip jit-lock when input is pending.
> AFAIK jit-lock is never skipped: it's (part of) the redisplay which is
> skipped (and that in turn may include calls to jit-lock).

This sounds like splitting hairs; if the part containing it is skipped, 
then jit-lock is skipped as well, obviously.

Does that happen when fast-but-imprecise-scrolling is nil as well?

Or is the situation that redisplay skips jit-lock when there's pending 
input (a good thing), but scrolling commands go back and force it 
anyway, to compute the new positions to scroll to?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > The mechanism it has now is effective only on fast machines.

  > Do you have any evidence to back that claim?

Others have posted the data here.  It is not crucial that I find it
myself.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Yes, that is what I meant.  The "kind of things" is nothing more than
  > writing comments in C (etc.) source code.  Sooner or later, such a
  > comment has a parenthesis in it, and sooner or later (e.g. by M-q), this
  > will end up in column 0,

I am surprised, because that never happens to me.  I don't put a * at
the start of each comment line, but I do indent each comment line
and that should be enough to make sure no openparen fills into column 0.

What do you think of such indentation as a solution for this problem.

  > My position at the moment is that the o-p-i-c-0-i-d-s mechanism should
  > remain active, but that we should fix the bugs in it such that the
  > backward scanning would only stop at a column-0 paren not in a comment
  > or string.  I outlined one way of doing this in the thread called "A
  > proposal for the future of open-paren-in-column-0-is-defun-start", but
  > nobody has responded to the thrust of this post at all, as yet.

If this can be done, I don't think I would argue against it.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>>> that = skip jit-lock when input is pending.
>> AFAIK jit-lock is never skipped: it's (part of) the redisplay which is
>> skipped (and that in turn may include calls to jit-lock).
> This sounds like splitting hairs; if the part containing it is skipped, then
> jit-lock is skipped as well, obviously.
>
> Does that happen when fast-but-imprecise-scrolling is nil as well?

Skipping redisplay when there's pending input happens regardless of
`fast-but-imprecise-scrolling`.

Now here's where "splitting hairs" becomes important: while the choice
to skip redisplay is not affected by `fast-but-imprecise-scrolling`, the
choice to skip jit-lock *is*.

> Or is the situation that redisplay skips jit-lock when there's pending input
> (a good thing), but scrolling commands go back and force it anyway, to
> compute the new positions to scroll to?

That's right (tho, AFAIK the scrolling commands cause jit-lock *before*
redisplay happens, so "go back" is not quite accurate).


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>   > > The mechanism it has now is effective only on fast machines.
>   > Do you have any evidence to back that claim?
> Others have posted the data here.

Which data are you referring to?  Maybe I missed it as well?

The only data I've seen so far that shows any effect of
open-paren-in-column-0 is the one recently (as in "after your above
messages") posted by Alan, and this data says nothing about slow or fast
machines, shows a relatively small performance difference (not
negligible, but within a factor 2), and considers a situation expected
to be rather rare.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>   > Yes, that is what I meant.  The "kind of things" is nothing more than
>   > writing comments in C (etc.) source code.  Sooner or later, such a
>   > comment has a parenthesis in it, and sooner or later (e.g. by M-q), this
>   > will end up in column 0,
>
> I am surprised, because that never happens to me.

Of course that never happens to you: you're the one who introduced the
open-paren-in-column-0 heuristic, so you're clearly biased.

> What do you think of such indentation as a solution for this problem.

That presumes you have control over the code (and coding conventions).


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >> Or, alternatively, (setq fast-but-imprecise-scrolling t). This var seems
 >> like a good idea in general, so we might consider going further with it.
 >
 > Try this:
 >
 > diff --git a/src/window.c b/src/window.c
 > index 8cdad27b66..b23ae41aba 100644
 > --- a/src/window.c
 > +++ b/src/window.c
 > @@ -5475,7 +5475,9 @@ window_scroll (Lisp_Object window, EMACS_INT n, bool whole, bool noerror)
 >
 >     wset_redisplay (XWINDOW (window));
 >
 > -  if (whole && Vfast_but_imprecise_scrolling)
 > +  if (Vfast_but_imprecise_scrolling
 > +      && (whole
 > +          || EQ (Vthis_command, Qmwheel_scroll)))
 >       specbind (Qfontification_functions, Qnil);
 >
 >     /* On GUI frames, use the pixel-based version which is much slower
 > @@ -8173,6 +8175,7 @@ syms_of_window (void)
 >     DEFSYM (Qmode_line_format, "mode-line-format");
 >     DEFSYM (Qheader_line_format, "header-line-format");
 >     DEFSYM (Qtab_line_format, "tab-line-format");
 > +  DEFSYM (Qmwheel_scroll, "mwheel-scroll");
 >
 >     DEFVAR_LISP ("temp-buffer-show-function", Vtemp_buffer_show_function,
 >   	       doc: /* Non-nil means call as function to display a help buffer.
 >
 >
 > This eliminates all the delay I formerly saw while mouse scrolling as
 > fast as possibly.

I tried that now and it makes no noticeable difference here.  Maybe I
did something wrong - all I checked via GDB was that it does perform the
specbinding.  Maybe that's not sufficient.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >>    > Recent Emacsen either ignore that variable or silently reset it to nil
 >>    > internally so it doesn't get into their way.  Their progmodes either
 >>    > always scan an entire buffer from its beginning or use some elaborate,
 >>    > fragile techniques to find such a top level position.  Moreover, our
 >>    > underlying mechanism for syntax highlighting always marks the entire
 >>    > rest of a buffer as dirty after every single editing change.  This has
 >>    > the consequence that that entire part has to be continuously rescanned
 >>    > when some of it is shown in another window.
 >>
 >> Does anyone disagree with this specific factual claim?
 >
 > I'm not sure what is "the claim" here, but I want to point out a small
 > inaccuracy: redisplay doesn't "continuously rescan the entire rest of
 > the buffer", it only rescans the part(s) shown in windows.

My text didn't say that _redisplay_ has to do that.  It talks about a
"consequence" and here the consequence is that jit-lock triggers an
eventual rescan of the rest of the buffer down to the part that is
displayed in another window.

 > (It might
 > happen that some major mode's font-lock definitions end up rescanning
 > much more, but that's a separate issue.)  And frankly, what would we
 > like Emacs to do instead?  A change in a buffer can potentially affect
 > the fontification of the rest of the buffer, and I don't think we
 > would like Emacs to fail to update other windows showing the same
 > buffer -- that would be against Emacs's description as "real-time"
 > editor.

It depends on what we consider a failure: I do not think that inserting
a "/*" in my window showing the top of my C buffer should cause the
window showing the bottom of my buffer consider everything part of a
comment starting at that "/*".  I think that the comment should stop at
the first paren in column zero after that "/*" even if that paren is
shown in the window showing the top of the buffer and even if that's
syntactically wrong (and hence the compiler would see it differently).

So what I would vote for in this case is that font-lock would specially
highlight the open paren that "stops seeing the comment as comment" so
the user is aware of the fact.  Would that be so hard to implement?
Even with a slight change in 'parse-partial-sexp' and 'syntax-ppss' that
would accompany it.  Optionally, by default turned off, so nobody who
wants the correct syntactic interpretation would be bothered.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Martin.

On Tue, Apr 07, 2020 at 10:31:08 +0200, martin rudalics wrote:
>  >> Or, alternatively, (setq fast-but-imprecise-scrolling t). This var seems
>  >> like a good idea in general, so we might consider going further with it.

>  > Try this:

>  > diff --git a/src/window.c b/src/window.c
>  > index 8cdad27b66..b23ae41aba 100644
>  > --- a/src/window.c
>  > +++ b/src/window.c
>  > @@ -5475,7 +5475,9 @@ window_scroll (Lisp_Object window, EMACS_INT n, bool whole, bool noerror)

>  >     wset_redisplay (XWINDOW (window));

>  > -  if (whole && Vfast_but_imprecise_scrolling)
>  > +  if (Vfast_but_imprecise_scrolling
>  > +      && (whole
>  > +          || EQ (Vthis_command, Qmwheel_scroll)))
>  >       specbind (Qfontification_functions, Qnil);

>  >     /* On GUI frames, use the pixel-based version which is much slower
>  > @@ -8173,6 +8175,7 @@ syms_of_window (void)
>  >     DEFSYM (Qmode_line_format, "mode-line-format");
>  >     DEFSYM (Qheader_line_format, "header-line-format");
>  >     DEFSYM (Qtab_line_format, "tab-line-format");
>  > +  DEFSYM (Qmwheel_scroll, "mwheel-scroll");

>  >     DEFVAR_LISP ("temp-buffer-show-function", Vtemp_buffer_show_function,
>  >   	       doc: /* Non-nil means call as function to display a help buffer.


>  > This eliminates all the delay I formerly saw while mouse scrolling as
>  > fast as possibly.

> I tried that now and it makes no noticeable difference here.  Maybe I
> did something wrong - all I checked via GDB was that it does perform
> the specbinding.  Maybe that's not sufficient.

That's not so good.  The way it works is by bypassing the fontification
in the scrolling code (which is only there to calculate how many lines to
scroll, particularly for variable height fonts).  The only fontification
should happen for redisplay, and that only for when there's no input
event waiting.

So if all works well, only isolated small segments of the buffer
(screenfuls) get fontified and displayed and no long queue of buffer
portions for fontification should build up.

I don't know why it's not working for you.

> martin

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

Hi Martin,

On 07.04.2020 11:31, martin rudalics wrote:
> I tried that now and it makes no noticeable difference here.  Maybe I
> did something wrong - all I checked via GDB was that it does perform the
> specbinding.  Maybe that's not sufficient.

Did you both apply the patch and set fast-but-imprecise-scrolling to t?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 07.04.2020 15:07, Dmitry Gutov wrote:
> Hi Martin,
> 
> On 07.04.2020 11:31, martin rudalics wrote:
>> I tried that now and it makes no noticeable difference here.  Maybe I
>> did something wrong - all I checked via GDB was that it does perform the
>> specbinding.  Maybe that's not sufficient.
> 
> Did you both apply the patch and set fast-but-imprecise-scrolling to t?

Sorry, nevermind. If the specbinding was applied, then you did.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >> I tried that now and it makes no noticeable difference here.  Maybe I
 >> did something wrong - all I checked via GDB was that it does perform
 >> the specbinding.  Maybe that's not sufficient.
 >
 > That's not so good.  The way it works is by bypassing the fontification
 > in the scrolling code (which is only there to calculate how many lines to
 > scroll, particularly for variable height fonts).  The only fontification
 > should happen for redisplay, and that only for when there's no input
 > event waiting.
 >
 > So if all works well, only isolated small segments of the buffer
 > (screenfuls) get fontified and displayed and no long queue of buffer
 > portions for fontification should build up.
 >
 > I don't know why it's not working for you.

As a matter of fact, it is slightly better.  Still, Emacs locks down
after a few scrolls in a similar way as without it.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > Did you both apply the patch and set fast-but-imprecise-scrolling to t?

Sure.  I also verified that

      specbind (Qfontification_functions, Qnil);

gets executed.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 07.04.2020 11:32, martin rudalics wrote:
> It depends on what we consider a failure: I do not think that inserting
> a "/*" in my window showing the top of my C buffer should cause the
> window showing the bottom of my buffer consider everything part of a
> comment starting at that "/*".  I think that the comment should stop at
> the first paren in column zero after that "/*" even if that paren is
> shown in the window showing the top of the buffer and even if that's
> syntactically wrong (and hence the compiler would see it differently).

To be accurate, the comment started by '/*' would end at the first 
occurrence of '*/' somewhere down the buffer.

If you insert or close a paren, however, that would definitely change 
syntax state at EOB.

The former case is possible to support better in syntax-ppss, FWIW. The 
latter we probably never will (in syntax-ppss).

> So what I would vote for in this case is that font-lock would specially
> highlight the open paren that "stops seeing the comment as comment" so
> the user is aware of the fact.  Would that be so hard to implement?
> Even with a slight change in 'parse-partial-sexp' and 'syntax-ppss' that
> would accompany it.  Optionally, by default turned off, so nobody who
> wants the correct syntactic interpretation would be bothered.

According to Alan's experiment, you can get this behavior now already 
simply by setting comment-use-syntax-ppss to nil and 
open-paren-in-column-0-is-defun-start to t.

Does it improve your experience significantly?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 07.04.2020 15:45, martin rudalics wrote:
>  > So if all works well, only isolated small segments of the buffer
>  > (screenfuls) get fontified and displayed and no long queue of buffer
>  > portions for fontification should build up.
>  >
>  > I don't know why it's not working for you.
> 
> As a matter of fact, it is slightly better.  Still, Emacs locks down
> after a few scrolls in a similar way as without it.

And can it similarly lock down for a long time? Or does it unfreeze quickly?

What about other major modes? If you create a big JS file and try the 
same experiment in js-mode? Or create a long Elisp file and do the same 
in emacs-lisp-mode (you can copy-paste a lot of time the contents of 
some existing file)? Or take any of those files, switch to 
fundamental-mode and do the same.

Does scrolling also lock up in a similar manner?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: rrandresf@gmail.com, emacs-devel@gnu.org, acm@muc.de
> From: martin rudalics <rudalics@gmx.at>
> Date: Tue, 7 Apr 2020 10:32:07 +0200
> 
>  > (It might
>  > happen that some major mode's font-lock definitions end up rescanning
>  > much more, but that's a separate issue.)  And frankly, what would we
>  > like Emacs to do instead?  A change in a buffer can potentially affect
>  > the fontification of the rest of the buffer, and I don't think we
>  > would like Emacs to fail to update other windows showing the same
>  > buffer -- that would be against Emacs's description as "real-time"
>  > editor.
> 
> It depends on what we consider a failure: I do not think that inserting
> a "/*" in my window showing the top of my C buffer should cause the
> window showing the bottom of my buffer consider everything part of a
> comment starting at that "/*".

If that is what the language syntax rules say, then yes, we should
paint everything in comment face.  If you disagree, then let's agree
to disagree: I consider "inaccurate" display one of the worst things
an editor can do to its users.

> So what I would vote for in this case is that font-lock would specially
> highlight the open paren that "stops seeing the comment as comment" so
> the user is aware of the fact.

How would that help if that parentheses is not shown in any window?
And even if it is shown, do we really expect users to pay attention to
that, and immediately understand that this parentheses with an unusual
face explains the wrong display?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: martin rudalics <rudalics@gmx.at>
> Date: Tue, 7 Apr 2020 14:45:42 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, rrandresf@gmail.com,
>  Dmitry Gutov <dgutov@yandex.ru>, rms@gnu.org, emacs-devel@gnu.org
> 
> As a matter of fact, it is slightly better.  Still, Emacs locks down
> after a few scrolls in a similar way as without it.

Maybe showing profiles for when Emacs locks up with and without that
setting could give some hints about what's going on?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 07.04.2020 02:41, Stefan Monnier wrote:
>> So you chose the absolute worst scenario for syntax-ppss, right? Where its
>> cache would be unavoidably clobbered at least once. I'm guessing the
>> difference between measurements is ~ how long (parse-partial-sexp
>> (point-min) (point-max)) takes on your machine in that file.
> 
> FWIW, `syntax-ppss` has not been optimized for this case at all, so
> I wouldn't be surprised if it takes noticeably more than
> 
>      (parse-partial-sexp (point-min) (point-max))
> 
> because it performs that computation in chunks, and the overhead for
> each chunk is likely non-negligible.

The times it takes here, in an optimized build, fluctuates from 0.02 to 
0.05. So indeed, the difference is larger. But the same order of 
magnitude, which is still pretty good, IMO.

>>> Here are the results:
>>>                            open-paren-in-column-zero-is-defun-start
>>> 			      nil                    t
>>> comment-use-syntax-ppss
>>>           nil                  0.319s                0.264s
>>> 	 t                   0.319s                0.227s
>>> .  Bearing in mind that c-u-s-p being t suppresses the action of
>>> o-p-i-c-0-i-d-s in back_comment, but not in beginning-of-defun, it seems
>>> like the o-p-i-c-0-i-d-s mechanism does indeed speed up some scenarios
>>> in C Mode, significantly but not massively.
>>> IMAO, It would be nice to have the code testing o-p-i-c-0-i-d-s (both
>>> places) able to ignore spurious cases of parens in literals.
> 
> The problem here is that in order to decide whether or not it's *still*
> spurious after the change near BOB, you basically have to compute the
> equivalent of `syntax-ppss`.  So we're back to square one.

That's a good point. In this particular scenario all column-1 parens 
down in the buffer should be invalidated.

> What could be done is to change `syntax-ppss` so it optimizes for the
> case where the buffer changes do not impact the way parsing is done
> "further down" (i.e. not for the case where the change just opened (or
> closed) a string/comment).  Not sure it's worth the trouble, OTOH.

Yup. Not sure either, but let's see what feedback others give.

Or we could try indefinitely supporting the "worse-is-better" semantics 
offered by o-p-i-c-0-i-d-s if, again, it's really in demand.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 6 Apr 2020 20:16:43 +0300
> 
> > What that binding does is affect the portions of display code that are
> > run as part of window_scroll_pixel_based, which the scroll command
> > calls to calculate a suitable position of window-start by simulating
> > display.  That display code runs with fontifications disabled, which
> > makes it faster.
> > 
> > All the scroll commands in Emacs do is more or less calculate the
> > window-start point; the rest is done when redisplay kicks in.
> 
> So what happens then? Does it trigger jit-lock anyway?

Basically, yes.  But with a twist, see below.

> If so, I don't understand how we get the apparent performance improvement from using fast-but-imprecise-scrolling.

To explain that, I need to step back and describe how jit-lock
mechanism works, from the display engine POV, and then how the scroll
commands work.  (See also a large commentary in keyboard.c around line
250.)

For simplicity, let's assume we have a buffer with no overlays, no
display strings, and no other complications, only faces.  Let's also
assume the text is entirely L2R, so we can ignore what bidi.c does.

When the display engine starts working on displaying a window, it
begins from the window-start position, loads all the faces and other
display-related stuff, then starts layout by moving one character at a
time in the direction of increasing buffer positions, and computing
the metrics of each character as determined by the face(s) at each
position.  It stops when it gets to a buffer position where the
'fontified' text property changes its value.  If the new value of
'fontified' is nil, the display engine calls fontification-functions.
jit-lock adds its function to that hook, so jit-lock is called.  What
jit-lock does, in a nutshell, is call the current buffer's
fontification function(s) on a chunk of text starting at the position
for which it was called, and it puts on that chunk a value of t for
the 'fontified' text property.  (Where that chunk ended is the
position where the 'fontified' property's value will again change to
nil, and if that position is inside the window, jit-lock will be
called again when the display engine gets to that position.)  After
fontification-functions return, the display engine processes the faces
in the chunk that has been fontified; then it continues marching
through the buffer.

Next, I need to describe how scroll commands work.  Assuming you typed
C-v, the scroll command computes the new window-start point.  It does
so by beginning at the current window-start, then moving through the
buffer text until it reaches the point that is N pixels down, where N
is the height of the window.  To move through the text, the scroll
command uses functions from the display engine that "simulate"
display, i.e. they perform the same layout calculations, but don't
prepare anything to be sent to the glass.  However -- and this is the
important part -- this "display simulation" uses the same code
described above that looks for positions where the 'fontified'
property is nil and calls fontification-functions to fontify such
text.  This is necessary, because fontification can change the metrics
of the text on display, and so any pixel-wise calculations and
decisions whether given text is or isn't inside a window must be sure
the text is fontified and has the right faces in order to produce
accurate results.

When the scroll command is done computing the new window-start point,
it sets a special flag telling redisplay to obey that window-start
value (otherwise redisplay might decide to compute window-start on its
own), and returns.

Then redisplay kicks in, and does its thing described above.  It will
find the entire text until the new window-start already fontified, and
also some text after the new window-start (because jit-lock works in
chunks, and usually fontifies slightly more than strictly needed).  So
displaying the new contents of the window will need to fontify only
those parts visible in the window that were not already fontified when
the scroll command looked for the new window-start.

If you lean on C-v, then the scroll command is repeatedly called to
find the next window-start, and repeatedly fontifies the text it goes
through.  If you scroll N screen-fulls, the fontification of the first
N-1 screen-fulls (if they were never displayed before, which happens
when you visit a file and immediately start scrolling) is "wasted
effort" for the purposes of the final window redisplay, because those
N-1 screen-fulls will not be displayed.  Only the fontification of the
last screen-full is "useful", because it saves some, sometimes all, of
the fontification job of the stuff to be eventually displayed when you
stop scrolling.

Enter fast-but-imprecise-scrolling.  What it does is disable jit-lock
during the part where the scroll command walked the buffer to find the
next window-start position.  Crucially, it also disables putting the
non-nil 'fontified' property on the text traversed by the scroll
command (because that is done by jit-lock).  So when you stop
scrolling, redisplay of the stuff that ends up in the window will have
to fontify all of what is visible in the window.

Bottom line: fast-but-imprecise-scrolling saves the work of jit-lock
when scrolling more than one screen-full, and is more-or-less useless
when scrolling one screen-full.  IOW, it is a "solution" for the use
case of leaning on C-v or M-v to scroll far into the buffer that was
never before displayed.

Of course, if fast-but-imprecise-scrolling makes the job of scroll
commands so much easier that Emacs is capable of keeping up with the
keyboard auto-repeat rate, then every screen-full you scroll through
_will_ be displayed, and therefore will be fontified by the calls to
jit-lock from redisplay itself -- and that will again make scrolling
slower.  So fast-but-imprecise-scrolling is most efficient when it
makes scroll commands faster, but not too fast...

Sorry for the long email, I hope this tricky issue is now at least a
tad more clear.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: rudalics@gmx.at, acm@muc.de, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 6 Apr 2020 21:59:29 +0300
> 
> Would I be correct to assume that you see only see this in CC Mode based 
> major mode, and only when editing big enough files? Such as xdisp.c 
> mentioned many times recently.

The former is probably correct.  But not the latter: there are some
constructs in C that make CC Mode notoriously slow, even in much
smaller buffers.  And just to make "notoriously slow" quantifiable: I
mean cases where typing RET takes half a second or a second.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 07.04.2020 17:30, Eli Zaretskii wrote:
>> Cc: rudalics@gmx.at, acm@muc.de, rrandresf@gmail.com, rms@gnu.org,
>>   emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Mon, 6 Apr 2020 21:59:29 +0300
>>
>> Would I be correct to assume that you see only see this in CC Mode based
>> major mode, and only when editing big enough files? Such as xdisp.c
>> mentioned many times recently.
> 
> The former is probably correct.  But not the latter: there are some
> constructs in C that make CC Mode notoriously slow, even in much
> smaller buffers.  And just to make "notoriously slow" quantifiable: I
> mean cases where typing RET takes half a second or a second.

In that case, there are several issues at play.

Scenarios that are made slower if the file is large are probably caused 
by the same architectural decisions that make (find-file "src/xdisp.c") 
slow-ish. All those could likely be improved if CC Mode migrated to 
using syntax-propertize-function, with its on-demand approach to parsing.

Scenarios which are slow no matter what probably deserve a separate bug 
report each, and are likely fixable without significant redesign.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Dmitry.

On Tue, Apr 07, 2020 at 18:04:13 +0300, Dmitry Gutov wrote:

> In that case, there are several issues at play.

> Scenarios that are made slower if the file is large are probably caused 
> by the same architectural decisions that make (find-file "src/xdisp.c") 
> slow-ish. All those could likely be improved if CC Mode migrated to 
> using syntax-propertize-function, with its on-demand approach to parsing.

Please would you stop saying this.  It's not true, and it gets very
wearing.  The crude syntax-propertize-function mechanism would have to
be considerably enhanced before it could support CC Mode's features.
And even then, it would be unlikely to be faster than the current CC
Mode.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 7 Apr 2020 18:04:13 +0300
> Cc: rudalics@gmx.at, rrandresf@gmail.com, emacs-devel@gnu.org, rms@gnu.org,
>  acm@muc.de
> 
> Scenarios which are slow no matter what probably deserve a separate bug 
> report each

Agreed, let's get that ball rolling.

> and are likely fixable without significant redesign.

Unfortunately, I'm not as optimistic as you on that one.

So here's the first bug report about a random slowness in CC Mode:

  . visit src/w32.c on the emacs-27 branch and go to line 8747
  . C-e DEL -- this deletes the last 'e' of "else"
  . the following line gets immediately fontified as a function name
  . type 'e' -- then watch how the fontification of the next line to
    its original face takes almost 1 sec -- this is the problem

FTR, this is in Emacs 27.0.90 built with -Og, but a -O2 compilation of
Emacs 26.3 takes only slightly less time in this case.

(I will file a formal bug report about this if Alan wants me to.)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

Hi Alan,

On 07.04.2020 20:42, Alan Mackenzie wrote:

>> Scenarios that are made slower if the file is large are probably caused
>> by the same architectural decisions that make (find-file "src/xdisp.c")
>> slow-ish. All those could likely be improved if CC Mode migrated to
>> using syntax-propertize-function, with its on-demand approach to parsing.
> 
> Please would you stop saying this.  It's not true, and it gets very
> wearing.  The crude syntax-propertize-function mechanism would have to
> be considerably enhanced before it could support CC Mode's features.

I will shut up about it now (saying it twice it plenty), but I am pretty 
confident saying that if you manage to migrate to s-p-f, file opening 
time will go down. Performance while typing is likely to improve too, at 
least when the same buffer is not shown in another window, many thousand 
lines later. It's not a given, but I'd really like to see you try 
collaborating with Stefan. At least for a prototype, to have a better 
idea of the associated performance tradeoffs. He posted a preliminary 
patch in the past, FWIW.

"Considerable enhancement" can also be a part of that discussion.

> And even then, it would be unlikely to be faster than the current CC
> Mode.

Some scenarios can become slower, that's for sure. But the more common 
ones can get faster. We won't know until we try.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 07.04.2020 22:22, Eli Zaretskii wrote:

>> and are likely fixable without significant redesign.
> 
> Unfortunately, I'm not as optimistic as you on that one.

In any case, s-p-f is unlikely to help there. ¯\_(ツ)_/¯

> So here's the first bug report about a random slowness in CC Mode:

Thank you.

>    . visit src/w32.c on the emacs-27 branch and go to line 8747
>    . C-e DEL -- this deletes the last 'e' of "else"
>    . the following line gets immediately fontified as a function name
>    . type 'e' -- then watch how the fontification of the next line to
>      its original face takes almost 1 sec -- this is the problem

I can and can't reproduce this at the same time.

Meaning, I do see "fontification of the next line to its original face" 
take some time. But if I don't wait for it to turn black again and 
continue typing, nothing bad happens: Emacs continues reacting to input 
just as promptly, and the next line turns black soon enough.

And if I type e.g. 'e RET', or 'e SPC', the next line turns black 
seemingly instantly.

> FTR, this is in Emacs 27.0.90 built with -Og, but a -O2 compilation of
> Emacs 26.3 takes only slightly less time in this case.

'emacs -Q' and a '-Og -g3' build of 27.0.90 here.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Eli.

On Tue, Apr 07, 2020 at 22:22:43 +0300, Eli Zaretskii wrote:
> > From: Dmitry Gutov <dgutov@yandex.ru>
> > Date: Tue, 7 Apr 2020 18:04:13 +0300
> > Cc: rudalics@gmx.at, rrandresf@gmail.com, emacs-devel@gnu.org, rms@gnu.org,
> >  acm@muc.de

> > Scenarios which are slow no matter what probably deserve a separate bug 
> > report each

> Agreed, let's get that ball rolling.

> > and are likely fixable without significant redesign.

> Unfortunately, I'm not as optimistic as you on that one.

> So here's the first bug report about a random slowness in CC Mode:

>   . visit src/w32.c on the emacs-27 branch and go to line 8747
>   . C-e DEL -- this deletes the last 'e' of "else"
>   . the following line gets immediately fontified as a function name
>   . type 'e' -- then watch how the fontification of the next line to
>     its original face takes almost 1 sec -- this is the problem

OK, I'm on Line 8759, where I've got:

  else
    cp->status = STATUS_READ_FAILED;

(I haven't updated for a few days.)  Is this the same place?

> FTR, this is in Emacs 27.0.90 built with -Og, but a -O2 compilation of
> Emacs 26.3 takes only slightly less time in this case.

Is this not jit-lock-context-time?

It takes 0.5s on my optimised build.  j-l-context-time is 0.5 here.
I'll admit I can't change the delay by changing j-l-c-t, but that might
be due to not reinitialising jit-lock properly.

However, first setting font-lock-support-mode to nil, then
reinitialising font-lock, deleting then reinserting that "e", the line
after the else _never_ gets its (lack of) face back.

> (I will file a formal bug report about this if Alan wants me to.)

No, it's fine like this!

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Dmitry.

On Tue, Apr 07, 2020 at 22:45:25 +0300, Dmitry Gutov wrote:
> Hi Alan,

> On 07.04.2020 20:42, Alan Mackenzie wrote:

> >> Scenarios that are made slower if the file is large are probably caused
> >> by the same architectural decisions that make (find-file "src/xdisp.c")
> >> slow-ish. All those could likely be improved if CC Mode migrated to
> >> using syntax-propertize-function, with its on-demand approach to parsing.

> > Please would you stop saying this.  It's not true, and it gets very
> > wearing.  The crude syntax-propertize-function mechanism would have to
> > be considerably enhanced before it could support CC Mode's features.

> I will shut up about it now (saying it twice it plenty), but I am pretty 
> confident saying that if you manage to migrate to s-p-f, file opening 
> time will go down.

I'm sure it would.  If file opening time were really a concern, a hybrid
algorithm would perhaps be the best way: apply the text properties first
in a lazy fashion, and thereafter treat them with care, as CC Mode
currently does.  But this would merely transfer the start up time to the
time taken in early scrolls forward.

> Performance while typing is likely to improve too, at least when the
> same buffer is not shown in another window, many thousand lines later.

What makes you think this?  Anyway, the case you cite has to be handled
well, too.

> It's not a given, but I'd really like to see you try collaborating
> with Stefan. At least for a prototype, to have a better idea of the
> associated performance tradeoffs. He posted a preliminary patch in the
> past, FWIW.

> "Considerable enhancement" can also be a part of that discussion.

The syntax-propertize-function mechanism works by erasing ALL
syntax-table properties after a change point, then reapplying them
lazily.  Considering that s-t properties have an overwhelmingly local
effect, this is very wasteful of processor time.

Consider, for example, editing within a large C++ raw string, a common
occurrence.  You yourself reported as a bug sluggish performance here in
mid 2016.  The cause was erasing too many s-t text properties at a
buffer change.  I think we were talking about 1 second per typed
character in the scenario you gave.  There are typically lots of these
properties in a raw string, in particular on " characters.

Consider(2) a C++ template: excusing my C++ syntax knowledge, type in

   template class foo < bar, baz >= bar>

, perhaps typing in the odd newline inside the template (a common
occurrence), or nesting further templates inside it (also a common
occurrence).  Note how the parenthesis text properties are added and
removed as you type.  All these modification are necessary, and they are
largely _before_ the point of insertion, not after it.

Splatting all the properties after the point of change would be
unnecessary and wasteful.

> > And even then, it would be unlikely to be faster than the current CC
> > Mode.

> Some scenarios can become slower, that's for sure. But the more common 
> ones can get faster. We won't know until we try.

Trying would be a _lot_ of work.  How is one to handle the common
example scenarios above?  Well, you'd have to enhance the
syntax-propertize-function with a means of determining a start position
for erasing s-t props, and also a stop position.  Once you do that,
you're effectively doing what CC Mode currently does, so where's the
speed advantage coming from?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > 
  > >   > Recent Emacsen either ignore that variable or silently reset it to nil
  > >   > internally so it doesn't get into their way.  Their progmodes either
  > >   > always scan an entire buffer from its beginning or use some elaborate,
  > >   > fragile techniques to find such a top level position.  Moreover, our
  > >   > underlying mechanism for syntax highlighting always marks the entire
  > >   > rest of a buffer as dirty after every single editing change.  This has
  > >   > the consequence that that entire part has to be continuously rescanned
  > >   > when some of it is shown in another window.
  > > 
  > > Does anyone disagree with this specific factual claim?

  > I'm not sure what is "the claim" here,

The factual statements in the paragraph I quoted, above.

					   but I want to point out a small
  > inaccuracy: redisplay doesn't "continuously rescan the entire rest of
  > the buffer", it only rescans the part(s) shown in windows.	(It might
  > happen that some major mode's font-lock definitions end up rescanning
  > much more, but that's a separate issue.)

Isn't that separate issue the issue we are talking about.

					      And frankly, what would we
  > like Emacs to do instead?

It would scan only from the last open-paren-in-column-zero, as it did
in the past.

			       A change in a buffer can potentially affect
  > the fontification of the rest of the buffer, and I don't think we
  > would like Emacs to fail to update other windows showing the same
  > buffer

I have a feeling there has been a change of topic here, but I can't be
sure.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >   > Yes, that is what I meant.  The "kind of things" is nothing more than
  > >   > writing comments in C (etc.) source code.  Sooner or later, such a
  > >   > comment has a parenthesis in it, and sooner or later (e.g. by M-q), this
  > >   > will end up in column 0,
  > >
  > > I am surprised, because that never happens to me.

  > Of course that never happens to you: you're the one who introduced the
  > open-paren-in-column-0 heuristic, so you're clearly biased.

Please focus on the issue not on personalities.  Anyone can be mistaken,
but don't argue that an objective claim is wrong based on who made it.

What matters is whether one can get good results from
open-paren-in-column-0, good enough to make it usable and useful.

I found the open-paren-in-column-0 heuristic satisfactory because in
practice I saw that open-parens did not drift into column zero.  Other
people have said that this does happen to them, and they can't avoid
it.

They, and I, were using the same editor, with the same heuristic, and
getting totally different results.  Why is that?  Surely that is pertinent.
I conjecture it is because of indentation conventions, but if that is not
the reason, what is?

  > > What do you think of such indentation as a solution for this problem.

  > That presumes you have control over the code (and coding conventions).

It presumes that the developers adopt the practice of indenting the
text of block comments, or can be persuaded to do so.

IF my conjecture is correct, open-paren-in-column-0 is useful for
programs where block comments are indented.  Is that conjecture
correct?  How about if we finish comparing notes, and see.

If that is how things are, it should be useful to support
open-paren-in-column-0-is-defun-start = t rather than make it a no-op.
If the code you edit does not indent block comments, you would
be happier with open-paren-in-column-0-is-defun-start = nil,
but you don't need that feature to be no-op'ed.

(I did not start indenting the text of block comments because of
the open-paren-in-column-0 heuristic.  I did it because that looks nice.)

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

You've written an extremely clear explanation of how fontification and
scrolling interact.  I'm impressed.

  > Of course, if fast-but-imprecise-scrolling makes the job of scroll
  > commands so much easier that Emacs is capable of keeping up with the
  > keyboard auto-repeat rate, then every screen-full you scroll through
  > _will_ be displayed, and therefore will be fontified by the calls to
  > jit-lock from redisplay itself -- and that will again make scrolling
  > slower.  So fast-but-imprecise-scrolling is most efficient when it
  > makes scroll commands faster, but not too fast...

I think that is true, if "faster" and "slower" are interpreted in
terms of CPU time used.  But if all of that work, including the
fontification of each screenful, is so quick that Emacs never falls
behind, I think it won't seem like "slowness" to the user -- who never
has to _wait_ for it.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> I found the open-paren-in-column-0 heuristic satisfactory because in
> practice I saw that open-parens did not drift into column zero.

Yes, it was a good heuristic when we needed one and when a heuristic was
good enough.  Nowadays it's not good enough any more.  Users who
experience a misbehavior because some "stray" open-paren in column-0
don't think "oh silly me, I shouldn't do that".  Instead they think
"Duh!  Emacs is stupid!".

> It presumes that the developers adopt the practice of indenting the
> text of block comments, or can be persuaded to do so.

What about those who can't or don't want to?

Should Emacs provide poor behavior when open parens in column 0 occur
for non-defuns?  Should Emacs provide poor performance when the buffer
doesn't have any open-parens in column 0 (as is the common case in many
languages)?

For these reasons, we came up with a different approach, not based on
a heuristic.  To make it work sufficiently efficiently it relies
on caching.  It's not always as efficient as the old heuristic, but on
the average it is more efficient, in the worst case it's a bit less
efficient, and in the best case it's a lot more efficient.

OK, I've wasted enough time in this thread, so I'll stop here.

I just recommend anyone who wants to hack on this problem to first do
some careful profiling (both at the Elisp level and the C level).
Then see if maybe the performance improvement that they're aiming for
can't be obtained simply by optimizing syntax-ppss (e.g. it's a pretty
simple code so it would not be hard to move it to C, if needed).

Finally, I'd urge anyone who's not willing to spend this effort to stop
arguing here about what they think should be done, because the likelihood
that the profiler would prove them wrong is just much too high.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Date: Tue, 7 Apr 2020 20:10:18 +0000
> Cc: Dmitry Gutov <dgutov@yandex.ru>, rudalics@gmx.at, rrandresf@gmail.com,
>   emacs-devel@gnu.org, rms@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> OK, I'm on Line 8759, where I've got:
> 
>   else
>     cp->status = STATUS_READ_FAILED;
> 
> (I haven't updated for a few days.)  Is this the same place?

Yup.

> > FTR, this is in Emacs 27.0.90 built with -Og, but a -O2 compilation of
> > Emacs 26.3 takes only slightly less time in this case.
> 
> Is this not jit-lock-context-time?

I have no idea.  But if you are saying that this is "normal", then I
disagree: I don't see why the change in fontification after DEL is
instantaneous, while the change after typing 'e' takes such annoyingly
long time.  Is jit-lock-context-time not involved in the former? if
so, why?

> However, first setting font-lock-support-mode to nil, then
> reinitialising font-lock, deleting then reinserting that "e", the line
> after the else _never_ gets its (lack of) face back.

So this is still rather a mystery, isn't it?

Thanks.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 7 Apr 2020 23:05:04 +0300
> Cc: rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org, acm@muc.de,
>  emacs-devel@gnu.org
> 
> I can and can't reproduce this at the same time.
> 
> Meaning, I do see "fontification of the next line to its original face" 
> take some time. But if I don't wait for it to turn black again and 
> continue typing, nothing bad happens: Emacs continues reacting to input 
> just as promptly, and the next line turns black soon enough.

Whether this is "bad" depends on what you use the fontification for.
For me, it's a sign of syntax correctness (among other things), so
waiting for the confirmation of the syntax for such a relatively long
time is quite "bad".

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: rudalics@gmx.at, rrandresf@gmail.com, emacs-devel@gnu.org,
> 	acm@muc.de
> Date: Tue, 07 Apr 2020 22:29:21 -0400
> 
>   > >   > Recent Emacsen either ignore that variable or silently reset it to nil
>   > >   > internally so it doesn't get into their way.  Their progmodes either
>   > >   > always scan an entire buffer from its beginning or use some elaborate,
>   > >   > fragile techniques to find such a top level position.  Moreover, our
>   > >   > underlying mechanism for syntax highlighting always marks the entire
>   > >   > rest of a buffer as dirty after every single editing change.  This has
>   > >   > the consequence that that entire part has to be continuously rescanned
>   > >   > when some of it is shown in another window.
> 
> 					   but I want to point out a small
>   > inaccuracy: redisplay doesn't "continuously rescan the entire rest of
>   > the buffer", it only rescans the part(s) shown in windows.	(It might
>   > happen that some major mode's font-lock definitions end up rescanning
>   > much more, but that's a separate issue.)
> 
> Isn't that separate issue the issue we are talking about.

I'm not sure.  The text says "scan an entire buffer" and "the entire
part has to be continuously rescanned", which could be interpreted as
though this rescanning is the main culprit for the slowness.  I wanted
to point out that these aspects are not forced by the infrastructure
that causes the displayed text to be fontified just in time, but by
some of the major mode's fontification functions called by the
infrastructure.  It is IMO important to understand where to look for
code that needs to be sped up, to avoid wasting effort on optimizing
places that don't contribute significantly to the slowdown.

> 					      And frankly, what would we
>   > like Emacs to do instead?
> 
> It would scan only from the last open-paren-in-column-zero, as it did
> in the past.

That would produce incorrect display, as I explained later:

> 			       A change in a buffer can potentially affect
>   > the fontification of the rest of the buffer, and I don't think we
>   > would like Emacs to fail to update other windows showing the same
>   > buffer
> 
> I have a feeling there has been a change of topic here, but I can't be
> sure.

I surely didn't mean to change the topic.  If the argument was that
stopping considering the fontification at some arbitrary point in the
buffer will improve performance, my point was that it OTOH might
produce inaccurate display, which I consider not a good idea.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: dgutov@yandex.ru, acm@muc.de, rrandresf@gmail.com,
> 	emacs-devel@gnu.org, rudalics@gmx.at
> Date: Tue, 07 Apr 2020 22:29:30 -0400
> 
> You've written an extremely clear explanation of how fontification and
> scrolling interact.  I'm impressed.

Thanks.  I thought it was important to explain this in so much detail
so that everyone interested in this issue will be on the same page.
What happens in this use case is quite complex, and many people don't
know about these details and therefore are very surprised by some
effects of some changes.

>   > Of course, if fast-but-imprecise-scrolling makes the job of scroll
>   > commands so much easier that Emacs is capable of keeping up with the
>   > keyboard auto-repeat rate, then every screen-full you scroll through
>   > _will_ be displayed, and therefore will be fontified by the calls to
>   > jit-lock from redisplay itself -- and that will again make scrolling
>   > slower.  So fast-but-imprecise-scrolling is most efficient when it
>   > makes scroll commands faster, but not too fast...
> 
> I think that is true, if "faster" and "slower" are interpreted in
> terms of CPU time used.

The CPU time is one major factor, of course.  But not the only one,
because the keyboard auto-repeat rate is a separate factor which can
be controlled and changed independently of the CPU speed.

> But if all of that work, including the fontification of each
> screenful, is so quick that Emacs never falls behind, I think it
> won't seem like "slowness" to the user -- who never has to _wait_
> for it.

That's true, to some extent, but my point was that the elapsed time
will become slower when each screenful is actually displayed while
scrolling.  It isn't an important point, though, just a side comment.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Eli.

On Wed, Apr 08, 2020 at 09:15:49 +0300, Eli Zaretskii wrote:
> > Date: Tue, 7 Apr 2020 20:10:18 +0000
> > Cc: Dmitry Gutov <dgutov@yandex.ru>, rudalics@gmx.at, rrandresf@gmail.com,
> >   emacs-devel@gnu.org, rms@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > OK, I'm on Line 8759, where I've got:

> >   else
> >     cp->status = STATUS_READ_FAILED;

> > (I haven't updated for a few days.)  Is this the same place?

> Yup.

> > > FTR, this is in Emacs 27.0.90 built with -Og, but a -O2 compilation of
> > > Emacs 26.3 takes only slightly less time in this case.

> > Is this not jit-lock-context-time?

> I have no idea.  But if you are saying that this is "normal", then I
> disagree: I don't see why the change in fontification after DEL is
> instantaneous, while the change after typing 'e' takes such annoyingly
> long time.  Is jit-lock-context-time not involved in the former? if
> so, why?

Is the problem more the asymmetry between the deletion and the
insertion than the actual time taken after the reinsertion?  We've had
context fontification since Emacs 21.

I admit I don't remember in detail why after the deletion the font
locking region is extended, but not after the reinsertion.  Possibly in
the former case, context fontification failed to do the job right.

> > However, first setting font-lock-support-mode to nil, then
> > reinitialising font-lock, deleting then reinserting that "e", the line
> > after the else _never_ gets its (lack of) face back.

> So this is still rather a mystery, isn't it?

It's context fontification.  I'm quite sure of that, now.  To restart
jit-lock with a new j-l-context-time involves explicitly getting rid of
a timer and allowing it to restart.

> Thanks.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Date: Wed, 8 Apr 2020 07:02:15 +0000
> Cc: dgutov@yandex.ru, rudalics@gmx.at, rrandresf@gmail.com,
>   emacs-devel@gnu.org, rms@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > Is this not jit-lock-context-time?
> 
> > I have no idea.  But if you are saying that this is "normal", then I
> > disagree: I don't see why the change in fontification after DEL is
> > instantaneous, while the change after typing 'e' takes such annoyingly
> > long time.  Is jit-lock-context-time not involved in the former? if
> > so, why?
> 
> Is the problem more the asymmetry between the deletion and the
> insertion than the actual time taken after the reinsertion?

Not the asymmetry, the slow refontifciation in the latter case.  It
won't help me if both refontifciations were "symmetrically" slow.

> > > However, first setting font-lock-support-mode to nil, then
> > > reinitialising font-lock, deleting then reinserting that "e", the line
> > > after the else _never_ gets its (lack of) face back.
> 
> > So this is still rather a mystery, isn't it?
> 
> It's context fontification.  I'm quite sure of that, now.  To restart
> jit-lock with a new j-l-context-time involves explicitly getting rid of
> a timer and allowing it to restart.

Then perhaps the solution is to reduce the default value of
jit-lock-context-time?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Date: Wed, 08 Apr 2020 10:10:07 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: rudalics@gmx.at, rrandresf@gmail.com, emacs-devel@gnu.org, rms@gnu.org,
>  dgutov@yandex.ru
> 
> > It's context fontification.  I'm quite sure of that, now.  To restart
> > jit-lock with a new j-l-context-time involves explicitly getting rid of
> > a timer and allowing it to restart.
> 
> Then perhaps the solution is to reduce the default value of
> jit-lock-context-time?

Btw, I cannot wrap my head around this "deferred context
fontification" stuff; never could.

The doc string of jit-lock-contextually says:

  If t, means fontification occurs on those lines modified and all
  subsequent lines.  This means those subsequent lines are refontified to reflect
  their new syntactic context, after `jit-lock-context-time' seconds.
  If any other value, e.g., `syntax-driven', means syntactically true
  fontification occurs only if syntactic fontification is performed using the
  buffer mode's syntax table, i.e., only if `font-lock-keywords-only' is nil.

I understand what t means and does, but what does non-nil, non-t value
do, and how is it different from t? in particular, does it also wait for
jit-lock-context-time of idleness?

The doc string of jit-lock-mode also seems to document only the t
value:

  - Deferred context fontification if `jit-lock-contextually' is
    non-nil.  This means fontification updates the buffer corresponding to
    true syntactic context, after `jit-lock-context-time' seconds of Emacs
    idle time, while Emacs remains idle.  Otherwise, fontification occurs
    on modified lines only, and subsequent lines can remain fontified
    corresponding to previous syntactic contexts.  This is useful where
    strings or comments span lines.

The code doesn't seem to start the jit-lock-context-timer if the value
is non-t, or am I missing something?  Since the default value is
'syntax-driven', where does that 0.5-second delay we all see come
from?  I guess you (Alan) have this figured out, so please share what
you've learned, and let's update the documentation to benefit from
that.

Thanks.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > To be accurate, the comment started by '/*' would end at the first occurrence of '*/' somewhere down the buffer.

If it appears before the open brace in column zero, yes.

 > If you insert or close a paren, however, that would definitely change syntax state at EOB.

It would still only get as far as to the next brace in column zero.

 > According to Alan's experiment, you can get this behavior now already simply by setting comment-use-syntax-ppss to nil and open-paren-in-column-0-is-defun-start to t.
 >
 > Does it improve your experience significantly?

No.  Fast and imprecise scrolling doesn't improve anything either here.
I'm surprised how people get improved behavior from these.  But I agree
with Stefan to shut up here from now on.  I don't have the time to
conduct an in-depth analysis of this problem and spotting a possible
culprit without such an analysis is, in fact, futile.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >> As a matter of fact, it is slightly better.  Still, Emacs locks down
 >> after a few scrolls in a similar way as without it.
 >
 > And can it similarly lock down for a long time? Or does it unfreeze quickly?

It doesn't lock down always.  But when it locks down it takes about the
same time to recover.

 > What about other major modes? If you create a big JS file and try the
 > same experiment in js-mode? Or create a long Elisp file and do the
 > same in emacs-lisp-mode (you can copy-paste a lot of time the contents
 > of some existing file)? Or take any of those files, switch to
 > fundamental-mode and do the same.

I don't use JS so I have no idea how to experiment with it.  Scrolling a
large Elisp file never locks down Emacs.  But note that the problem with
scrolling happens already with dispextern.h whose size is about a tenth
of that of xdisp.c.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >> It depends on what we consider a failure: I do not think that inserting
 >> a "/*" in my window showing the top of my C buffer should cause the
 >> window showing the bottom of my buffer consider everything part of a
 >> comment starting at that "/*".
 >
 > If that is what the language syntax rules say, then yes, we should
 > paint everything in comment face.  If you disagree, then let's agree
 > to disagree: I consider "inaccurate" display one of the worst things
 > an editor can do to its users.

Turning the tables, I don't consider inaccurate editing (like inserting
an unbalanced paren or delimiter) one of the worst things a user can do
to its editor.  For me, an open paren in column zero is some sort of an
agreement between me and the editor we both would heed as accurately as
we can.  People like you who don't care would not even pay attention to
it.  People like me who do care might profit from it.  It certainly
isn't a cure of all problems and might not even help with scrolling
buffers.  But it would provide one knob a user could press to find out.

 >> So what I would vote for in this case is that font-lock would specially
 >> highlight the open paren that "stops seeing the comment as comment" so
 >> the user is aware of the fact.
 >
 > How would that help if that parentheses is not shown in any window?

Why bother?  The comment would appear highlighted as comment down to
that parenthesis or to the end of the window, whatever comes first.

 > And even if it is shown, do we really expect users to pay attention to
 > that, and immediately understand that this parentheses with an unusual
 > face explains the wrong display?

I certainly would.  In my last commit I did insert such a paren on line
8022 of xfns.c and didn't notice it.  OTOH, the average user who never
customized 'open-paren-in-column-0-is-defun-start' would not be aware of
it.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > It stops when it gets to a buffer position where the
 > 'fontified' text property changes its value.  If the new value of
 > 'fontified' is nil, the display engine calls fontification-functions.
 > jit-lock adds its function to that hook, so jit-lock is called.  What
 > jit-lock does, in a nutshell, is call the current buffer's
 > fontification function(s) on a chunk of text starting at the position
 > for which it was called, and it puts on that chunk a value of t for
 > the 'fontified' text property.  (Where that chunk ended is the
 > position where the 'fontified' property's value will again change to
 > nil, and if that position is inside the window, jit-lock will be
 > called again when the display engine gets to that position.)  After
 > fontification-functions return, the display engine processes the faces
 > in the chunk that has been fontified; then it continues marching
 > through the buffer.

IIRC fontification can also modify text before the position where the
'fontified' property changed to nil.  Or is that wrong?

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >> Is this not jit-lock-context-time?
 >
 > I have no idea.  But if you are saying that this is "normal", then I
 > disagree: I don't see why the change in fontification after DEL is
 > instantaneous, while the change after typing 'e' takes such annoyingly
 > long time.  Is jit-lock-context-time not involved in the former? if
 > so, why?

For me that behavior is "normal" in the sense that I see it happen here
all the time, when editing C files, on the average once in a minute.
But since it never disrupts my typing I simply got used to it.

What's bothering me more is that when I put an overlay somewhere in a
buffer and eventually remove it, underlying text properties provided by
the font locking mechanism are never restored unless I explicitly edit
the line where such properties should be seen.  Apparently so, because
font locking does not trigger when overlays get removed because that
does not count as a buffer change.  Another thing I got used to ...

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: dgutov@yandex.ru, rrandresf@gmail.com, emacs-devel@gnu.org, rms@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Wed, 8 Apr 2020 10:38:30 +0200
> 
> What's bothering me more is that when I put an overlay somewhere in a
> buffer and eventually remove it, underlying text properties provided by
> the font locking mechanism are never restored unless I explicitly edit
> the line where such properties should be seen.  Apparently so, because
> font locking does not trigger when overlays get removed because that
> does not count as a buffer change.

Overlay changes do count as buffer changes, for the purposes of
redisplay, so a bug report with a recipe will be appreciated.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 08.04.2020 11:36, martin rudalics wrote:
> I don't use JS so I have no idea how to experiment with it.  Scrolling a
> large Elisp file never locks down Emacs.

OK, thanks. That does point to CC Mode, rather than some general display 
engine problems.

> But note that the problem with
> scrolling happens already with dispextern.h whose size is about a tenth
> of that of xdisp.c.

Yeah, scrolling performance shouldn't be affected by the size of the 
file anyway. It's only editing performance that could.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: Eli Zaretskii <eliz@gnu.org>, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 8 Apr 2020 13:37:08 +0300
> 
> > But note that the problem with
> > scrolling happens already with dispextern.h whose size is about a tenth
> > of that of xdisp.c.
> 
> Yeah, scrolling performance shouldn't be affected by the size of the 
> file anyway.

Scrolling performance _can_ be affected by the size of the buffer
(especially if the major mode's fontifications look far back), but
only when you scroll close to its end.  When you start scrolling from
the beginning, all buffers will behave similarly.  I think what martin
says is that the problem starts happening almost immediately, so being
some relatively short distance from the file's beginning already
produces the slowdown.  Which then will affect smaller files as well,
of course.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rrandresf@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Wed, 8 Apr 2020 10:37:46 +0200
> 
>  > It stops when it gets to a buffer position where the
>  > 'fontified' text property changes its value.  If the new value of
>  > 'fontified' is nil, the display engine calls fontification-functions.
>  > jit-lock adds its function to that hook, so jit-lock is called.  What
>  > jit-lock does, in a nutshell, is call the current buffer's
>  > fontification function(s) on a chunk of text starting at the position
>  > for which it was called, and it puts on that chunk a value of t for
>  > the 'fontified' text property.  (Where that chunk ended is the
>  > position where the 'fontified' property's value will again change to
>  > nil, and if that position is inside the window, jit-lock will be
>  > called again when the display engine gets to that position.)  After
>  > fontification-functions return, the display engine processes the faces
>  > in the chunk that has been fontified; then it continues marching
>  > through the buffer.
> 
> IIRC fontification can also modify text before the position where the
> 'fontified' property changed to nil.  Or is that wrong?

What is actually fontified depends on the buffer's
fontification-functions; jit-lock just calls those functions with a
region that starts at the first position in the window that has its
'fontified' property nil, and ends jit-lock-chunk-size (500 bytes by
default) after that.  However, the 'fontified' property will be set by
jit-lock to t only for text in that chunk, as jit-lock has no way of
knowing what was actually fontified.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 08.04.2020 14:22, Eli Zaretskii wrote:
> Scrolling performance_can_  be affected by the size of the buffer
> (especially if the major mode's fontifications look far back)

That's true, of course, but we have various options to avoid that, so 
wouldn't expect this particular failing from a mature major mode package.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> The doc string of jit-lock-contextually says:
>
>   If t, means fontification occurs on those lines modified and all
>   subsequent lines.  This means those subsequent lines are refontified to reflect
>   their new syntactic context, after `jit-lock-context-time' seconds.
>   If any other value, e.g., `syntax-driven', means syntactically true
>   fontification occurs only if syntactic fontification is performed using the
>   buffer mode's syntax table, i.e., only if `font-lock-keywords-only' is nil.
>
> I understand what t means and does, but what does non-nil, non-t value
> do, and how is it different from t?

This is a global variable.  The non-nil non-t value means "treat it as
nil in some buffers and as t in others, depending on
`font-lock-keywords-only`" (well, more or less: other users of jit-lock
than font-lock can also affect it, but as a first approximation that's
about it).

> in particular, does it also wait for jit-lock-context-time
> of idleness?

Yes.

> The code doesn't seem to start the jit-lock-context-timer if the value
> is non-t, or am I missing something?

Yes, you're missing:

    (when (and contextual jit-lock-contextually)
      (setq-local jit-lock-contextually t))

which will turn the non-nil non-t value into t when applicable.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> What's bothering me more is that when I put an overlay somewhere in a
> buffer and eventually remove it, underlying text properties provided by
> the font locking mechanism are never restored

Overlays should usually not affect the way font-lock *puts* properties.
So when you remove the overlay, there should be no need to re-run
font-lock, and the mere redisplay should make the underlying text
properties (previously installed by font-lock) visible.

But all this depends on specifics, so without knowing what kind of
overlay you're putting/removing when/where/how it's hard to know where's
the problem.

IOW: make it a bug report.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> IIRC fontification can also modify text before the position where the
> 'fontified' property changed to nil.

It can do basically anything it likes, so yes, that can happen.
That can also cause the display to be incorrect because the redisplay
has already rendered that text.

> Or is that wrong?

It's not wrong, but if you *need* it, then you may have to do extra
work to make sure it's handled correctly.

E.g. jit-lock clients can tell jit-lock which part of the buffer they
actually fontified by returning a value of the form
`(jit-lock-bounds START . END)`.  START can then find its way to the
following code (under the name `loose-beg`):

               ;; The redisplay engine has already rendered the buffer up-to
               ;; `orig-start' and won't notice if the above jit-lock-functions
               ;; changed the appearance of any part of the buffer prior
               ;; to that.  So if `loose-beg' is before `orig-start', we need to
               ;; cause a new redisplay cycle after this one so that the changes
               ;; are properly reflected on screen.
               ;; To make such repeated redisplay happen less often, we can
               ;; eagerly extend the refontified region with
               ;; jit-lock-after-change-extend-region-functions.
               (when (< loose-beg orig-start)
                 (run-with-timer 0 nil #'jit-lock-force-redisplay
                                 (copy-marker loose-beg)
                                 (copy-marker orig-start)))


-- Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: acm@muc.de,  rudalics@gmx.at,  rrandresf@gmail.com,
>   emacs-devel@gnu.org,  rms@gnu.org,  dgutov@yandex.ru
> Date: Wed, 08 Apr 2020 10:22:29 -0400
> 
> > I understand what t means and does, but what does non-nil, non-t value
> > do, and how is it different from t?
> 
> This is a global variable.  The non-nil non-t value means "treat it as
> nil in some buffers and as t in others, depending on
> `font-lock-keywords-only`" (well, more or less: other users of jit-lock
> than font-lock can also affect it, but as a first approximation that's
> about it).
> 
> > in particular, does it also wait for jit-lock-context-time
> > of idleness?
> 
> Yes.

Thanks.

> > The code doesn't seem to start the jit-lock-context-timer if the value
> > is non-t, or am I missing something?
> 
> Yes, you're missing:
> 
>     (when (and contextual jit-lock-contextually)
>       (setq-local jit-lock-contextually t))
> 
> which will turn the non-nil non-t value into t when applicable.

This is in jit-lock-register, which is not called from jit-lock.el.
Does this mean that non-nil, non-t value of jit-lock-contextually is
like nil, unless the major mode calls jit-lock-register with its 2nd
argument non-nil?  And if so, why does the doc string mention
font-lock-keywords-only and doesn't mention jit-lock-register?  They
seem to be at least theoretically independent?  If not, where's their
connection?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> This is in jit-lock-register, which is not called from jit-lock.el.
> Does this mean that non-nil, non-t value of jit-lock-contextually is
> like nil, unless the major mode calls jit-lock-register with its 2nd
> argument non-nil?

Yes.

> And if so, why does the doc string mention font-lock-keywords-only and
> doesn't mention jit-lock-register?

I think that's because that's written for the end-user rather than for
the programmer: the end user doesn't know `jit-lock-register`, but it
may know `font-lock-keywords-only`.

But maybe it's really just an artifact of history (originally, jit-lock
and font-lock were more tightly linked).

> They seem to be at least theoretically independent?  If not, where's
> their connection?

`font-lock-keywords-only` ends up passed by font-lock as
2nd arg to `jit-lock-register`.


        Stefan


PS: BTW, maybe `jit-lock-contextually` should really be part of
font-lock rather than jit-lock.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: acm@muc.de,  rudalics@gmx.at,  rrandresf@gmail.com,
>   emacs-devel@gnu.org,  rms@gnu.org,  dgutov@yandex.ru
> Date: Wed, 08 Apr 2020 11:16:29 -0400
> 
> > This is in jit-lock-register, which is not called from jit-lock.el.
> > Does this mean that non-nil, non-t value of jit-lock-contextually is
> > like nil, unless the major mode calls jit-lock-register with its 2nd
> > argument non-nil?
> 
> Yes.
> 
> > And if so, why does the doc string mention font-lock-keywords-only and
> > doesn't mention jit-lock-register?
> 
> I think that's because that's written for the end-user rather than for
> the programmer: the end user doesn't know `jit-lock-register`, but it
> may know `font-lock-keywords-only`.
> 
> But maybe it's really just an artifact of history (originally, jit-lock
> and font-lock were more tightly linked).
> 
> > They seem to be at least theoretically independent?  If not, where's
> > their connection?
> 
> `font-lock-keywords-only` ends up passed by font-lock as
> 2nd arg to `jit-lock-register`.

Thanks, I tried to clarify the docs using this information.

RE: emacs rendering comparisson between emacs23 and emacs26.3

[Drew Adams]

> Whether this is "bad" depends on what you use the fontification for.
> For me, it's a sign of syntax correctness (among other things), so
> waiting for the confirmation of the syntax for such a relatively long
> time is quite "bad".

I don't mean to distract from the discussion.

But what you say there seems useful to me: syntax
highlighting can serve more than one purpose.

Given that, could one solution/approach perhaps be
to let users choose (e.g. by variable, possibly
buffer-local) the behavior they want in this regard?

For example, some users, or users in some contexts
(e.g. some modes), might prefer to sacrifice some
degree of fidelity (correctness) between syntax and
fontification, if that provides better performance
in those contexts.

Again, if this question/comment is a distraction,
feel free to ignore.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > It presumes that the developers adopt the practice of indenting the
  > > text of block comments, or can be persuaded to do so.

  > What about those who can't or don't want to?

Provided they do not set open-paren-in-column-0-is-defun-start to t,
nothing will change for them.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > It would scan only from the last open-paren-in-column-zero, as it did
  > > in the past.

  > That would produce incorrect display, as I explained later:

  > > 			       A change in a buffer can potentially affect
  > >   > the fontification of the rest of the buffer, and I don't think we
  > >   > would like Emacs to fail to update other windows showing the same
  > >   > buffer

I could not see how that relates to the issue.  That point was not
explicitly stated.  That's why I said it seemed like a change in
topic.

Now you've explained how they relate, so now I understand your point.

I am suggesting that we once again support a mode in which users
signal the start of a defun with an open-delimiter in column zero.
For CC modes, this would have to be explicitly enabled.

If you are editing code written in the usual GNU style 
then you can enable that mode and get good results.

If you are editing code NOT written in that style
then don't enable the mode.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: acm@muc.de, rudalics@gmx.at, eliz@gnu.org, rrandresf@gmail.com,
> 	emacs-devel@gnu.org
> Date: Wed, 08 Apr 2020 23:10:06 -0400
> 
>   > > It presumes that the developers adopt the practice of indenting the
>   > > text of block comments, or can be persuaded to do so.
> 
>   > What about those who can't or don't want to?
> 
> Provided they do not set open-paren-in-column-0-is-defun-start to t,
> nothing will change for them.

I think there might be a miscommunication here.  I think Stefan
alluded to editing source files outside of Emacs (more generally,
outside the GNU Project), where the style is different and not under
any control by us.  Those source files could violate this convention,
and we will not be able to do anything about that, except adapt Emacs
to handle them reasonably well.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >   > > It presumes that the developers adopt the practice of indenting the
  > >   > > text of block comments, or can be persuaded to do so.
  > > 
  > >   > What about those who can't or don't want to?
  > > 
  > > Provided they do not set open-paren-in-column-0-is-defun-start to t,
  > > nothing will change for them.

  > I think there might be a miscommunication here.  I think Stefan
  > alluded to editing source files outside of Emacs (more generally,
  > outside the GNU Project), where the style is different and not under
  > any control by us.

I think so too.  I responded to that by saying people should not turn on
this mode for those files:

  > > Provided they do not set open-paren-in-column-0-is-defun-start to t,
  > > nothing will change for them.


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> I think so too.  I responded to that by saying people should not turn on
> this mode for those files:

Performance has to be be good in files without open-parens in column-0.
So setting this var *should* provide no significant performance benefit
(otherwise it means performance is not good enough in files without
open-parens in column-0 and we should fix that).

A setting that is not just a performance heuristic would be fine,
e.g. something that causes the syntax-ppss state to be forcibly reset to
"toplevel" whenever we bump into an open-paren in column 0 (which seems
to be something Martin wants, so that a non-closed comment doesn't leak
past an open-paren in column-0).  But that's a completely separate issue
(beside the superficial connection that they both have to do with parens
in column-0).

But this too long thread is about performance, and when it comes to
performance any heuristic based on open-parens in-column-0 is
a non-starter AFAIC because they only cover a subset of cases and
we need something that covers the other cases as well.
And AFAIK we already have such a "something", so there's no need to
settle for this half-solution, no matter how cheap it may be nor how
attached to it some people might be.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

Hi Alan,

On 08.04.2020 05:09, Alan Mackenzie wrote:

>> I will shut up about it now (saying it twice it plenty), but I am pretty
>> confident saying that if you manage to migrate to s-p-f, file opening
>> time will go down.
> 
> I'm sure it would.  If file opening time were really a concern, a hybrid
> algorithm would perhaps be the best way: apply the text properties first
> in a lazy fashion, and thereafter treat them with care, as CC Mode
> currently does.

s-p-f would help with the first step, and as for "treating them with 
care", it would be good for us all to see how much of that is really 
needed. Improving syntax-propertize is not out of the question, and it 
might benefit several major modes, not just the CC collection.

> But this would merely transfer the start up time to the
> time taken in early scrolls forward.

Not really. The start up scans the whole buffer, doesn't it? The early 
scrolls forward would still scan only a fraction of it.

>> Performance while typing is likely to improve too, at least when the
>> same buffer is not shown in another window, many thousand lines later.
> 
> What makes you think this?

Inserting characters can alter the syntax state of the whole buffer. At 
least that's true for some of them. Full buffer scan sounds inevitable 
in those cases.

>> "Considerable enhancement" can also be a part of that discussion.
> 
> The syntax-propertize-function mechanism works by erasing ALL
> syntax-table properties after a change point, then reapplying them
> lazily.

That's not right. It only erases syntax-table properties in a chunk 
before calling syntax-propertize-function on the same range of 
positions. IOW, is overwrites them lazily as well.

 > Considering that s-t properties have an overwhelmingly local
 > effect, this is very wasteful of processor time.

It would have been. As you can see, it's not a difficult problem to fix, 
even if it were still present.

> Consider, for example, editing within a large C++ raw string, a common
> occurrence.  You yourself reported as a bug sluggish performance here in
> mid 2016.  The cause was erasing too many s-t text properties at a
> buffer change.  I think we were talking about 1 second per typed
> character in the scenario you gave.  There are typically lots of these
> properties in a raw string, in particular on " characters.

I'm pretty sure I have thought of that example because it's an instance 
of a syntax problem that's easy enough to solve within 
syntax-propertize-function framework.

> Consider(2) a C++ template: excusing my C++ syntax knowledge, type in
> 
>     template class foo < bar, baz >= bar>
> 
> , perhaps typing in the odd newline inside the template (a common
> occurrence), or nesting further templates inside it (also a common
> occurrence).  Note how the parenthesis text properties are added and
> removed as you type.  All these modification are necessary, and they are
> largely _before_ the point of insertion, not after it.

The current implementation of applying these properties can probably be 
transferred into a syntax-propertize-function with only modest changes.

>> Some scenarios can become slower, that's for sure. But the more common
>> ones can get faster. We won't know until we try.
> 
> Trying would be a _lot_ of work.  How is one to handle the common
> example scenarios above?

Stefan has offered to help. And I'm sure he could answer the follow-up 
questions much better than I.

> Well, you'd have to enhance the
> syntax-propertize-function with a means of determining a start position
> for erasing s-t props, and also a stop position.

The real-world uses of s-p-f out there already solve syntax problems of 
comparable complexity. And move the start position, among other things.

> Once you do that,
> you're effectively doing what CC Mode currently does, so where's the
> speed advantage coming from?

 From doing things more lazily, is how I see it. But I'm not an expert 
on CC Mode architecture.

Among other benefits, moving it to a standard-ish framework like s-p-f 
could (possibly) simplify its code, as well as make it more approachable 
for other developers already familiar with how most other major modes 
are written. So far I wouldn't even know where to start fixing bugs in 
it, and IMHO CC Mode currently has bus factor = 1. It's not great for 
its future. I suspect it's not ideal for you either.

Simply collaborating with one other developer on an overhaul project 
(whether it succeeds or not; perhaps partially) can improve on that.

   Cheers.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Performance has to be be good in files without open-parens in column-0.

We would certainly like that to be the case.  Is that the case now,
even on the machines we can use in the Free World?

If so, we don't have a problem.

If not, an improvement for GNU-formatted code is better than nothing.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> We would certainly like that to be the case.  Is that the case now,
> even on the machines we can use in the Free World?

In my experience on my good old T60 and T61, yes.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Noam Postavsky]

On Tue, 7 Apr 2020 at 04:45, martin rudalics <rudalics@gmx.at> wrote:

> So what I would vote for in this case is that font-lock would specially
> highlight the open paren that "stops seeing the comment as comment" so
> the user is aware of the fact.  Would that be so hard to implement?
> Even with a slight change in 'parse-partial-sexp' and 'syntax-ppss' that
> would accompany it.  Optionally, by default turned off, so nobody who
> wants the correct syntactic interpretation would be bothered.

I think Bug#21871 "Emacs Lisp Mode (at least): spurious parens in
column 0 don't get bold red highlighting." is related. Perhaps the
patch I posted at https://debbugs.gnu.org/21871#57 could help?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Dmitry.

On Fri, Apr 10, 2020 at 06:33:13 +0300, Dmitry Gutov wrote:
> Hi Alan,

> On 08.04.2020 05:09, Alan Mackenzie wrote:

> >> I will shut up about it now (saying it twice it plenty), but I am pretty
> >> confident saying that if you manage to migrate to s-p-f, file opening
> >> time will go down.

> > I'm sure it would.  If file opening time were really a concern, a hybrid
> > algorithm would perhaps be the best way: apply the text properties first
> > in a lazy fashion, and thereafter treat them with care, as CC Mode
> > currently does.

> s-p-f would help with the first step, and as for "treating them with 
> care", it would be good for us all to see how much of that is really 
> needed. Improving syntax-propertize is not out of the question, and it 
> might benefit several major modes, not just the CC collection.

> > But this would merely transfer the start up time to the
> > time taken in early scrolls forward.

> Not really. The start up scans the whole buffer, doesn't it? The early 
> scrolls forward would still scan only a fraction of it.

I'm thinking more about "scrolling" to the function in the file that one
wants to work on or look at.  On average, this will be a little more
than half way through the file (there is often a large comment block at
BOB).  So you'd only be saving about half of CC Mode's start-up scan.

> >> Performance while typing is likely to improve too, at least when the
> >> same buffer is not shown in another window, many thousand lines later.

> > What makes you think this?

> Inserting characters can alter the syntax state of the whole buffer. At 
> least that's true for some of them. Full buffer scan sounds inevitable 
> in those cases.

Full buffer scans are very unusual.  Inserting a " where every
subsequent line ended with a backslash might do that.  Inserting a C++ <
wouldn't - its effect is limited to up to the next brace or semicolon.

Inserting a C++ raw string opener does typically necessitate a full
scan (a search for a matching closer), but that would also be the case
using syntax-propertize.

> >> "Considerable enhancement" can also be a part of that discussion.

> > The syntax-propertize-function mechanism works by erasing ALL
> > syntax-table properties after a change point, then reapplying them
> > lazily.

> That's not right. It only erases syntax-table properties in a chunk 
> before calling syntax-propertize-function on the same range of 
> positions. IOW, is overwrites them lazily as well.

Sorry, I was mistaken there.  The bounds for erasing and re-applying the
s-t props are determined (except in simple cases) by
syntax-propertize-extend-region-functions.

So, we would merely be moving functions from
c-get-state-before-change-functions and c-before-font-lock-functions
(effectively lists of before-/after-change functions) to
s-p-extend-region-f, together with adaptation.  Would you agree that
such a change to CC Mode would be largely pointless if some of these
functions had to remain on c-get-state-b-c-f and c-before-f-l-f?

But the way s-p-extend-region-f functions are called is to keep calling
them repeatedly until they've all said "no change" together.  This would
dramatically slow down CC Mode, where currently these functions are each
called exactly once.

Also, the syntax-propertize mechanism is weaker than CC Mode's: When it
is run, there is no way of knowing whether it's being called as a change
function, and if it is, OLD-LEN is discarded.  How can it have access to
variables set in before-change-functions?  (An example of such is
c-raw-string-end-delim-disrupted.  In before change, it is set when the
existing raw string end delimiter is about to cease to be.  In after
change, the fact of this flag being nil means we don't need to search
for an alternative closing delimiter, etc.  This change can obviously
not be detected in an after-change function.)

>  > Considering that s-t properties have an overwhelmingly local
>  > effect, this is very wasteful of processor time.

> It would have been. As you can see, it's not a difficult problem to fix, 
> even if it were still present.

The lack of full information (see above) in the syntax-propertize
mechanism is a problem.

> > Consider, for example, editing within a large C++ raw string, a common
> > occurrence.  You yourself reported as a bug sluggish performance here in
> > mid 2016.  The cause was erasing too many s-t text properties at a
> > buffer change.  I think we were talking about 1 second per typed
> > character in the scenario you gave.  There are typically lots of these
> > properties in a raw string, in particular on " characters.

> I'm pretty sure I have thought of that example because it's an instance 
> of a syntax problem that's easy enough to solve within 
> syntax-propertize-function framework.

Having actually gone through all the issues and implemented raw strings,
I can't agree with you there.  There are all sorts of subtleties which
necessitate intimate cooperation between the before-change-functions and
after-change-functions.  Such cooperation seems to be excluded by the
syntax-propertize mechanism.

> > Consider(2) a C++ template: excusing my C++ syntax knowledge, type in

> >     template class foo < bar, baz >= bar>

> > , perhaps typing in the odd newline inside the template (a common
> > occurrence), or nesting further templates inside it (also a common
> > occurrence).  Note how the parenthesis text properties are added and
> > removed as you type.  All these modification are necessary, and they are
> > largely _before_ the point of insertion, not after it.

> The current implementation of applying these properties can probably be 
> transferred into a syntax-propertize-function with only modest changes.

Maybe, but with a slowdown.  More of these properties will get erased
than needed (with nested template forms), and they will all need to get
put back again.

> >> Some scenarios can become slower, that's for sure. But the more common
> >> ones can get faster. We won't know until we try.

Other than starting up a buffer, we still haven't identified any
specific scenarios where speed up might happen.

> > Trying would be a _lot_ of work.  How is one to handle the common
> > example scenarios above?

> Stefan has offered to help. And I'm sure he could answer the follow-up 
> questions much better than I.

I've tried quite a few optimisations over the years.  Some have been
successful, but all too often I've put in a lot of work, then at the end
of it the profiler tells me It's just been a waste of time.  I strongly
suspect that that would be the result here, too.

> > Well, you'd have to enhance the syntax-propertize-function with a
> > means of determining a start position for erasing s-t props, and
> > also a stop position.

> The real-world uses of s-p-f out there already solve syntax problems of 
> comparable complexity. And move the start position, among other things.

OK.  I was mistaken there.

> > Once you do that, you're effectively doing what CC Mode currently
> > does, so where's the speed advantage coming from?

>  From doing things more lazily, is how I see it. But I'm not an expert 
> on CC Mode architecture.

> Among other benefits, moving it to a standard-ish framework like s-p-f 
> could (possibly) simplify its code, as well as make it more approachable 
> for other developers already familiar with how most other major modes 
> are written. So far I wouldn't even know where to start fixing bugs in 
> it, and IMHO CC Mode currently has bus factor = 1. It's not great for 
> its future. I suspect it's not ideal for you either.

I don't think the syntax-propertize mechanism is all that brilliant.
It's too constrained, and places too many restrictions on what can be
done with the syntax-table text property.  For example, (from
syntax.el):

(defvar syntax-propertize-function nil
  ;; Rather than a -functions hook, this is a -function because it's easier
  ;; to do a single scan than several scans: with multiple scans, one cannot
  ;; assume that the text before point has been propertized, so syntax-ppss
  ;; gives unreliable results (and stores them in its cache to boot, so we'd
  ;; have to flush that cache between each function, and we couldn't use
  ;; syntax-ppss-flush-cache since that would not only flush the cache but also
  ;; reset syntax-propertize--done which should not be done in this case).

From my point of view, "multiple scans" are _much_ easier.  They are
prohibited here only because syntax-ppss and syntax-propertize-function
have got themselves tied up in a tight knot.  One answer would be not to
use syntax-ppss inside a s-p-function.  (CC Mode doesn't use syntax-ppss
at all).  Another answer would be to give the responsibility of removing
the s-t text properties to the s-p-function.

> Simply collaborating with one other developer on an overhaul project 
> (whether it succeeds or not; perhaps partially) can improve on that.

But take a massive amount of time.

>    Cheers.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

Hi Alan,

On 12.04.2020 18:34, Alan Mackenzie wrote:
>>> But this would merely transfer the start up time to the
>>> time taken in early scrolls forward.
> 
>> Not really. The start up scans the whole buffer, doesn't it? The early
>> scrolls forward would still scan only a fraction of it.
> 
> I'm thinking more about "scrolling" to the function in the file that one
> wants to work on or look at.  On average, this will be a little more
> than half way through the file (there is often a large comment block at
> BOB).  So you'd only be saving about half of CC Mode's start-up scan.

Yes, on average it's only 2x benefit, but then again, if the buffer 
opens at its beginning first, the extra initialization will be spread 
across user commands. Which should have some additional positive effect 
on apparent responsiveness.

>> Inserting characters can alter the syntax state of the whole buffer. At
>> least that's true for some of them. Full buffer scan sounds inevitable
>> in those cases.
> 
> Full buffer scans are very unusual.  Inserting a " where every
> subsequent line ended with a backslash might do that.  Inserting a C++ <
> wouldn't - its effect is limited to up to the next brace or semicolon.
> 
> Inserting a C++ raw string opener does typically necessitate a full
> scan (a search for a matching closer), but that would also be the case
> using syntax-propertize.

Not really. It would just mark the opener as a string opener (maybe with 
some extra text property), and that's that. Then font-lock would fontify 
the following text as string contents (until the end of the window or a 
little bit farther). Then you type the closer, it only has to scan a 
little far back (it'll call syntax-ppss to find the string opener), the 
closer is propertized as appropriate, and that's that. No full buffer 
scans at any step.

I recall that fontifying the rest of the buffer as text after a simple 
string opener could be a sore topic for you, but raw strings should be 
rare enough (aren't they?), or if they are not, fontification logic 
could opt to do something different, while syntax-table properties will 
be applied the "correct" way.

> So, we would merely be moving functions from
> c-get-state-before-change-functions and c-before-font-lock-functions
> (effectively lists of before-/after-change functions) to
> s-p-extend-region-f, together with adaptation. Would you agree that
> such a change to CC Mode would be largely pointless if some of these
> functions had to remain on c-get-state-b-c-f and c-before-f-l-f?

Yes, I think before-change-functions should become empty. Or much emptier.

> But the way s-p-extend-region-f functions are called is to keep calling
> them repeatedly until they've all said "no change" together.  This would
> dramatically slow down CC Mode, where currently these functions are each
> called exactly once.

Here's a simple solution: create one function special for CC Mode that 
would do that, and add it to s-p-extend-region-f.

But there might be even better way to do that. I'm not the best person 
to discuss that with.

> Also, the syntax-propertize mechanism is weaker than CC Mode's: When it
> is run, there is no way of knowing whether it's being called as a change
> function, and if it is, OLD-LEN is discarded.  How can it have access to
> variables set in before-change-functions?  (An example of such is
> c-raw-string-end-delim-disrupted.  In before change, it is set when the
> existing raw string end delimiter is about to cease to be.  In after
> change, the fact of this flag being nil means we don't need to search
> for an alternative closing delimiter, etc.  This change can obviously
> not be detected in an after-change function.)

As we seem to agree, before-change-functions should not be needed. 
Neither should be the code that tracks the details of the edits that the 
user makes. That alone can simplify some things.

>> I'm pretty sure I have thought of that example because it's an instance
>> of a syntax problem that's easy enough to solve within
>> syntax-propertize-function framework.
> 
> Having actually gone through all the issues and implemented raw strings,
> I can't agree with you there.  There are all sorts of subtleties which
> necessitate intimate cooperation between the before-change-functions and
> after-change-functions.  Such cooperation seems to be excluded by the
> syntax-propertize mechanism.

It encourages a different approach. Again: there are examples of raw 
strings support in other major modes.

>>> Consider(2) a C++ template: excusing my C++ syntax knowledge, type in
> 
>>>      template class foo < bar, baz >= bar>
> 
>>> , perhaps typing in the odd newline inside the template (a common
>>> occurrence), or nesting further templates inside it (also a common
>>> occurrence).  Note how the parenthesis text properties are added and
>>> removed as you type.  All these modification are necessary, and they are
>>> largely _before_ the point of insertion, not after it.
> 
>> The current implementation of applying these properties can probably be
>> transferred into a syntax-propertize-function with only modest changes.
> 
> Maybe, but with a slowdown.  More of these properties will get erased
> than needed (with nested template forms), and they will all need to get
> put back again.

We won't really know until we can measure the result.

>>>> Some scenarios can become slower, that's for sure. But the more common
>>>> ones can get faster. We won't know until we try.
> 
> Other than starting up a buffer, we still haven't identified any
> specific scenarios where speed up might happen.

When before-changes-functions only contains (t syntax-ppss-flush-cache), 
that can visibly change the performance tradeoffs.

> I don't think the syntax-propertize mechanism is all that brilliant.
> It's too constrained, and places too many restrictions on what can be
> done with the syntax-table text property.  For example, (from
> syntax.el):

I wouldn't say it's perfect either. But it's proven helpful over the 
years, and provided a base design for a lot of major mode implementations.

> (defvar syntax-propertize-function nil
>    ;; Rather than a -functions hook, this is a -function because it's easier
>    ;; to do a single scan than several scans: with multiple scans, one cannot
>    ;; assume that the text before point has been propertized, so syntax-ppss
>    ;; gives unreliable results (and stores them in its cache to boot, so we'd
>    ;; have to flush that cache between each function, and we couldn't use
>    ;; syntax-ppss-flush-cache since that would not only flush the cache but also
>    ;; reset syntax-propertize--done which should not be done in this case).
> 
>  From my point of view, "multiple scans" are _much_ easier.  They are
> prohibited here only because syntax-ppss and syntax-propertize-function
> have got themselves tied up in a tight knot.  One answer would be not to
> use syntax-ppss inside a s-p-function.  (CC Mode doesn't use syntax-ppss
> at all).  Another answer would be to give the responsibility of removing
> the s-t text properties to the s-p-function.

I think we could extend the customizability in that direction. But first 
we'd have to see clear evidence that the current design is not good 
enough (e.g. CC Mode has been reimplemented on top of it, and the result 
is decidedly not fast enough).

>> Simply collaborating with one other developer on an overhaul project
>> (whether it succeeds or not; perhaps partially) can improve on that.
> 
> But take a massive amount of time.

You could consider it an investment.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>   ;; Rather than a -functions hook, this is a -function because it's easier
>   ;; to do a single scan than several scans: with multiple scans, one cannot
[...]
> From my point of view, "multiple scans" are _much_ easier.

Obviously it's a matter of opinion.

Luckily, you have a Turing complete language at hand, so
`syntax-propertize-function` does not *impose* a single scan, it just
encourages it (especially with `syntax-propertize-rules`).

Your `syntax-propertize-function` can definitely use multiple scans if
it wants (and it's simply up to you to be careful about the fact that
until the last scan, `syntax-ppss` (and `parse-partial-sexp`) will
obviously not give the "final" answer, and if you use `syntax-ppss`
there, it's then your responsability to manually flush its cache with
`syntax-ppss-flush-cache`).

Really, you can use `syntax-propertize-function` with CC-mode without
significant changes to the code (i.e. reusing the code mostly unchanged
for the actual work).  The basic idea is to have CC-mode pretend that
the buffer only extends as far as `syntax-propertize--done`:

- when opening the file, pretend it's empty (i.e. do the initial scan
  as if `point-max` were equal to `point-min`).
- when `syntax-propertize-function` is called with arguments BEG and END,
  treat it as if the old code was faced with the following sequence of
  operations:
  1- delete all the text between BEG and `point-max`.
  2- insert at the end of file the new text found between BEG and END.

Clearly, the old code is supposed to handle correctly all three elements
(an empty file, deletion of text at EOB, and insertion of text at EOB),
so the old code will be able to handle those requests correctly without
any need for a redesign (you just have to "hide" the rest of the buffer
from the corresponding code, which might be doable with narrowing).

Of course the code of CC-mode is designed to handle many more operations
than deletion and insertion at EOB, so after that first step you could
try and adjust the code accordingly, but that can only make it simpler,
and is not necessary for a first cut.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Dmitry.

On Thu, Apr 16, 2020 at 04:43:23 +0300, Dmitry Gutov wrote:
> Hi Alan,

> On 12.04.2020 18:34, Alan Mackenzie wrote:
> >>> But this would merely transfer the start up time to the
> >>> time taken in early scrolls forward.

> >> Not really. The start up scans the whole buffer, doesn't it? The early
> >> scrolls forward would still scan only a fraction of it.

> > I'm thinking more about "scrolling" to the function in the file that one
> > wants to work on or look at.  On average, this will be a little more
> > than half way through the file (there is often a large comment block at
> > BOB).  So you'd only be saving about half of CC Mode's start-up scan.

> Yes, on average it's only 2x benefit, but then again, if the buffer 
> opens at its beginning first, the extra initialization will be spread 
> across user commands. Which should have some additional positive effect 
> on apparent responsiveness.

Will it?  A slightly faster appearance of the buffer at line 1 will be
offset by a slightly more sluggish response when doing M-x imenu foo.  And
"slightly" here means a fraction of a second on a modern machine,
presumably somewhat longer on an older one.

> >> Inserting characters can alter the syntax state of the whole buffer. At
> >> least that's true for some of them. Full buffer scan sounds inevitable
> >> in those cases.

> > Full buffer scans are very unusual.  Inserting a " where every
> > subsequent line ended with a backslash might do that.  Inserting a C++ <
> > wouldn't - its effect is limited to up to the next brace or semicolon.

> > Inserting a C++ raw string opener does typically necessitate a full
> > scan (a search for a matching closer), but that would also be the case
> > using syntax-propertize.

> Not really. It would just mark the opener as a string opener (maybe with 
> some extra text property), and that's that.

You don't know whether it's an unterminated raw string (the usual case)
until you've scanned for a potential closing delimiter.  This affects the
font locking.  (An unterminated opening delimiter gets
font-lock-warning-face, a terminated one doesn't.)

This is the sort of feature which I'm not willing to sacrifice.

> Then font-lock would fontify the following text as string contents
> (until the end of the window or a little bit farther). Then you type
> the closer, it only has to scan a little far back (it'll call
> syntax-ppss to find the string opener), the closer is propertized as
> appropriate, and that's that. No full buffer scans at any step.

> I recall that fontifying the rest of the buffer as text after a simple 
> string opener could be a sore topic for you, but raw strings should be 
> rare enough (aren't they?), or if they are not, fontification logic 
> could opt to do something different, while syntax-table properties will 
> be applied the "correct" way.

I'm not sure what you mean by "as text".  I've no reason to think raw
strings are at all rare.  I've had several bug reports for them.  I'm not
sure what you mean by "fontification logic ... something different" - do
you mean in the raw string case?  Or by "while s-t properties will be
applied in the "correct" way" - these properties are currently correctly
applied (modulo any remaining bugs).

> > So, we would merely be moving functions from
> > c-get-state-before-change-functions and c-before-font-lock-functions
> > (effectively lists of before-/after-change functions) to
> > s-p-extend-region-f, together with adaptation. Would you agree that
> > such a change to CC Mode would be largely pointless if some of these
> > functions had to remain on c-get-state-b-c-f and c-before-f-l-f?

> Yes, I think before-change-functions should become empty. Or much emptier.

It can't become empty.  after-change-functions is fine for dealing with
insertions, but can't do much after a deletion.  Consider the case where
you're in a string and all you know is that 5 characters have been
deleted.  Those characters might have been )foo", so after checking the
beginning of the string starts off with R"bar(, you've then got to scan a
long way forward looking for )bar".  Effectively every deletion within a
string would involve scanning to the end of that string.

By contrast, in a before-change-function, a quick check will show that
)foo", which is about to be deleted is here not a raw string delimiter,
so no special action is needed.

> > But the way s-p-extend-region-f functions are called is to keep calling
> > them repeatedly until they've all said "no change" together.  This would
> > dramatically slow down CC Mode, where currently these functions are each
> > called exactly once.

> Here's a simple solution: create one function special for CC Mode that 
> would do that, and add it to s-p-extend-region-f.

> But there might be even better way to do that. I'm not the best person 
> to discuss that with.

Sorry for not thinking it through, that's not a problem at all, really.

> > Also, the syntax-propertize mechanism is weaker than CC Mode's: When it
> > is run, there is no way of knowing whether it's being called as a change
> > function, and if it is, OLD-LEN is discarded.  How can it have access to
> > variables set in before-change-functions?  (An example of such is
> > c-raw-string-end-delim-disrupted.  In before change, it is set when the
> > existing raw string end delimiter is about to cease to be.  In after
> > change, the fact of this flag being nil means we don't need to search
> > for an alternative closing delimiter, etc.  This change can obviously
> > not be detected in an after-change function.)

> As we seem to agree, before-change-functions should not be needed. 

I don't agree at all.  before-change-functions is needed (see above).

> Neither should be the code that tracks the details of the edits that the 
> user makes. That alone can simplify some things.

At the cost of complicating other things, and slowing things down.  If
you have no b-c-f, you have no information about a deletion beyond its
OLD-LEN.  So you have to do a lot more checking and scanning in an a-c-f
to recover that lost information.

> >> I'm pretty sure I have thought of that example because it's an instance
> >> of a syntax problem that's easy enough to solve within
> >> syntax-propertize-function framework.

> > Having actually gone through all the issues and implemented raw strings,
> > I can't agree with you there.  There are all sorts of subtleties which
> > necessitate intimate cooperation between the before-change-functions and
> > after-change-functions.  Such cooperation seems to be excluded by the
> > syntax-propertize mechanism.

> It encourages a different approach. Again: there are examples of raw 
> strings support in other major modes.

Do any of them have the syntactical complexities of C++'s raw strings?

> >>> Consider(2) a C++ template: excusing my C++ syntax knowledge, type in

> >>>      template class foo < bar, baz >= bar>

> >>> , perhaps typing in the odd newline inside the template (a common
> >>> occurrence), or nesting further templates inside it (also a common
> >>> occurrence).  Note how the parenthesis text properties are added and
> >>> removed as you type.  All these modification are necessary, and they are
> >>> largely _before_ the point of insertion, not after it.

> >> The current implementation of applying these properties can probably be
> >> transferred into a syntax-propertize-function with only modest changes.

> > Maybe, but with a slowdown.  More of these properties will get erased
> > than needed (with nested template forms), and they will all need to get
> > put back again.

> We won't really know until we can measure the result.

What's the point in investing all the effort to make the change, when
there's not even a prediction of a speed up?

> >>>> Some scenarios can become slower, that's for sure. But the more common
> >>>> ones can get faster. We won't know until we try.

> > Other than starting up a buffer, we still haven't identified any
> > specific scenarios where speed up might happen.

> When before-changes-functions only contains (t syntax-ppss-flush-cache), 
> that can visibly change the performance tradeoffs.

It can indeed, for the worse.  See above.

> > I don't think the syntax-propertize mechanism is all that brilliant.
> > It's too constrained, and places too many restrictions on what can be
> > done with the syntax-table text property.  For example, (from
> > syntax.el):

> I wouldn't say it's perfect either. But it's proven helpful over the 
> years, and provided a base design for a lot of major mode implementations.

Your first sentence is the pertinent one, here.

And I'm not sure where the proof of the syntax-propertize mechanism being
helpful is.  Has anybody but its originator positively chosen to use it,
whilst being aware of the alternatives?

And the mechanism has caused difficult to debug bugs, such as bug #40661.

To become usable for CC Mode, it would need to provide something on
before-change-functions to complement what's on a-c-f, and it would need
to provide some control to the major mode over which syntax-table
properties get erased.

> > (defvar syntax-propertize-function nil
> >    ;; Rather than a -functions hook, this is a -function because it's easier
> >    ;; to do a single scan than several scans: with multiple scans, one cannot
> >    ;; assume that the text before point has been propertized, so syntax-ppss
> >    ;; gives unreliable results (and stores them in its cache to boot, so we'd
> >    ;; have to flush that cache between each function, and we couldn't use
> >    ;; syntax-ppss-flush-cache since that would not only flush the cache but also
> >    ;; reset syntax-propertize--done which should not be done in this case).

> >  From my point of view, "multiple scans" are _much_ easier.  They are
> > prohibited here only because syntax-ppss and syntax-propertize-function
> > have got themselves tied up in a tight knot.  One answer would be not to
> > use syntax-ppss inside a s-p-function.  (CC Mode doesn't use syntax-ppss
> > at all).  Another answer would be to give the responsibility of removing
> > the s-t text properties to the s-p-function.

> I think we could extend the customizability in that direction. But first 
> we'd have to see clear evidence that the current design is not good 
> enough (e.g. CC Mode has been reimplemented on top of it, and the result 
> is decidedly not fast enough).

Before we even start reimplementing CC Mode, we'd want some evidence that
the current design of syntax-propertize is adequate.  I see quite clearly
that it is not.

> >> Simply collaborating with one other developer on an overhaul project
> >> (whether it succeeds or not; perhaps partially) can improve on that.

> > But take a massive amount of time.

> You could consider it an investment.

As yet, I don't honestly see how.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 19.04.2020 20:12, Alan Mackenzie wrote:

>> Yes, on average it's only 2x benefit, but then again, if the buffer
>> opens at its beginning first, the extra initialization will be spread
>> across user commands. Which should have some additional positive effect
>> on apparent responsiveness.
> 
> Will it?  A slightly faster appearance of the buffer at line 1 will be
> offset by a slightly more sluggish response when doing M-x imenu foo.  And
> "slightly" here means a fraction of a second on a modern machine,
> presumably somewhat longer on an older one.

M-x file-file followed by M-x imenu should take ~as much as before. And 
if the user doesn't go for Imenu or scrolling to EOB right away, they 
get extra performance.

I don't know how slow it's for the average user, but there certainly are 
some with slow machines out there. Or people compiling without 
optimizations.

>>> Inserting a C++ raw string opener does typically necessitate a full
>>> scan (a search for a matching closer), but that would also be the case
>>> using syntax-propertize.
> 
>> Not really. It would just mark the opener as a string opener (maybe with
>> some extra text property), and that's that.
> 
> You don't know whether it's an unterminated raw string (the usual case)
> until you've scanned for a potential closing delimiter.

Is C++ syntax so ambiguous? Can R"( mean something else?

> This affects the
> font locking.  (An unterminated opening delimiter gets
> font-lock-warning-face, a terminated one doesn't.)

If everything after R"( is fontified as a string, it serves as a 
"warning" of sort as well.

> This is the sort of feature which I'm not willing to sacrifice.

Is it worth a full buffer scan every time you write a new raw string 
literal?

>> Then font-lock would fontify the following text as string contents
>> (until the end of the window or a little bit farther). Then you type
>> the closer, it only has to scan a little far back (it'll call
>> syntax-ppss to find the string opener), the closer is propertized as
>> appropriate, and that's that. No full buffer scans at any step.
> 
>> I recall that fontifying the rest of the buffer as text after a simple
>> string opener could be a sore topic for you, but raw strings should be
>> rare enough (aren't they?), or if they are not, fontification logic
>> could opt to do something different, while syntax-table properties will
>> be applied the "correct" way.
> 
> I'm not sure what you mean by "as text".

Sorry, I meant "as string".

> I've no reason to think raw
> strings are at all rare.  I've had several bug reports for them.  I'm not
> sure what you mean by "fontification logic ... something different" - do
> you mean in the raw string case?

I mean that if a raw string is unterminated, the default behavior should 
be to fontify the rest of the buffer as string. But then again, you 
could choose some different highliting in font-lock rules.

> Or by "while s-t properties will be
> applied in the "correct" way" - these properties are currently correctly
> applied (modulo any remaining bugs).

In that case, you should always be able to find the beginning of the 
current string literal (raw or plain) using syntax-ppss.

>> Yes, I think before-change-functions should become empty. Or much emptier.
> 
> It can't become empty.  after-change-functions is fine for dealing with
> insertions, but can't do much after a deletion.  Consider the case where
> you're in a string and all you know is that 5 characters have been
> deleted.  Those characters might have been )foo", so after checking the
> beginning of the string starts off with R"bar(, you've then got to scan a
> long way forward looking for )bar".  Effectively every deletion within a
> string would involve scanning to the end of that string.

This is an example of extra complexity you have to retain to implement 
the above feature the way you do.

It's probably also an example of how before/after-change-functions 
essentially duplicate the knowledge of language syntax. I'm guessing 
here, but to make it work like that, you need to have multiple functions 
"understand" the raw string syntax.

Whereas with syntax-propertize-function, that knowledge is concentrated 
in one place (maybe two, if font-lock rules do something unusual). This 
way, the code is simplified.

>>> Having actually gone through all the issues and implemented raw strings,
>>> I can't agree with you there.  There are all sorts of subtleties which
>>> necessitate intimate cooperation between the before-change-functions and
>>> after-change-functions.  Such cooperation seems to be excluded by the
>>> syntax-propertize mechanism.
> 
>> It encourages a different approach. Again: there are examples of raw
>> strings support in other major modes.
> 
> Do any of them have the syntactical complexities of C++'s raw strings?

You tell me:

https://docs.ruby-lang.org/en/2.4.0/syntax/literals_rdoc.html#label-Percent+Strings
https://rosettacode.org/wiki/Here_document#Ruby
https://github.com/atom/language-ruby/issues/109#issue-98715165

>>> Maybe, but with a slowdown.  More of these properties will get erased
>>> than needed (with nested template forms), and they will all need to get
>>> put back again.
> 
>> We won't really know until we can measure the result.
> 
> What's the point in investing all the effort to make the change, when
> there's not even a prediction of a speed up?

In principle, the speed-up will come from:

- Deferred execution (where several buffer changes can be handled 
together and not right away),
- No parsing the buffer contents much farther than the current window, 
in most cases. Which can speed up the majority of user actions. The 
exceptions will remain slower, but that is often a good tradeoff.

> And I'm not sure where the proof of the syntax-propertize mechanism being
> helpful is.  Has anybody but its originator positively chosen to use it,
> whilst being aware of the alternatives?

The alternatives being reinventing the relevant logic from zero in each 
major mode? And writing syntax caching logic each time?

> To become usable for CC Mode, it would need to provide something on
> before-change-functions to complement what's on a-c-f, and it would need
> to provide some control to the major mode over which syntax-table
> properties get erased.

Not something I can comment on.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>> You don't know whether it's an unterminated raw string (the usual case)
>> until you've scanned for a potential closing delimiter.
> Is C++ syntax so ambiguous? Can R"( mean something else?

No it has nothing to do with the language's syntax.

It's just a UI choice: Alan likes to highlight the opening string
delimiter of unterminated strings [ Not sure why CC-mode doesn't do the
same for unterminated comments, tho maybe it's just an oversight.  ]

I think this should not vary between major modes, IOW it should be
a separate feature that works on all major modes.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 20.04.2020 02:21, Stefan Monnier wrote:
>>> You don't know whether it's an unterminated raw string (the usual case)
>>> until you've scanned for a potential closing delimiter.
>> Is C++ syntax so ambiguous? Can R"( mean something else?
> No it has nothing to do with the language's syntax.
> 
> It's just a UI choice: Alan likes to highlight the opening string
> delimiter of unterminated strings [ Not sure why CC-mode doesn't do the
> same for unterminated comments, tho maybe it's just an oversight.  ]
> 
> I think this should not vary between major modes, IOW it should be
> a separate feature that works on all major modes.

If it can be done reliably outside of CC Mode (i.e. without relying on 
knowing the particulars of the buffer's programming language), then it's 
certainly another argument for using 
syntax-propertize-function/syntax-ppss framework.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> If it can be done reliably outside of CC Mode (i.e. without relying on
> knowing the particulars of the buffer's programming language),

It can be done reliably, but it would most likely want help from the
major mode (e.g. the C syntax tables give the syntax of strings but
don't indicate that those strings are not allowed to span multiple
lines).


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Stefan.

On Mon, Apr 20, 2020 at 15:01:14 -0400, Stefan Monnier wrote:
> > If it can be done reliably outside of CC Mode (i.e. without relying on
> > knowing the particulars of the buffer's programming language),

> It can be done reliably, but it would most likely want help from the
> major mode (e.g. the C syntax tables give the syntax of strings but
> don't indicate that those strings are not allowed to span multiple
> lines).

How about adding support to the syntax table?

For example, a new syntax class called something like "line string
quote".  One of these would begin a @def{line string}, and another one
or an unescaped NL (whichever came first) would end it.

Easy enough to implement, at least for forward scanning.  back_comment
might cause some fun.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Stefan.

On Sun, Apr 19, 2020 at 19:21:59 -0400, Stefan Monnier wrote:
> >> You don't know whether it's an unterminated raw string (the usual case)
> >> until you've scanned for a potential closing delimiter.
> > Is C++ syntax so ambiguous? Can R"( mean something else?

> No it has nothing to do with the language's syntax.

> It's just a UI choice: Alan likes to highlight the opening string
> delimiter of unterminated strings [ Not sure why CC-mode doesn't do the
> same for unterminated comments, tho maybe it's just an oversight.  ]

There isn't really such a thing as an unterminated comment.  A line
comment extends to the next unescaped EOL, a block comment extends to
the next */ or EOB.  The syntax routines and font locking already handle
it well.

> I think this should not vary between major modes, IOW it should be
> a separate feature that works on all major modes.


>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 20.04.2020 23:06, Alan Mackenzie wrote:
> A line
> comment extends to the next unescaped EOL, a block comment extends to
> the next */ or EOB.

And an unterminated raw string doesn't?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Dmitry.

On Sun, Apr 19, 2020 at 21:12:57 +0300, Dmitry Gutov wrote:
> On 19.04.2020 20:12, Alan Mackenzie wrote:

[ .... ]

> >>> Inserting a C++ raw string opener does typically necessitate a full
> >>> scan (a search for a matching closer), but that would also be the
> >>> case using syntax-propertize.

> >> Not really. It would just mark the opener as a string opener (maybe with
> >> some extra text property), and that's that.

> > You don't know whether it's an unterminated raw string (the usual case)
> > until you've scanned for a potential closing delimiter.

> Is C++ syntax so ambiguous? Can R"( mean something else?

C++ syntax is ambiguous in several places, but not here.  R"foo(,
assuming it's not itself inside a literal, means exactly one thing.  But
you don't know whether it starts an unterminated string unless you seach
for a closing delimiter.

> > This affects the font locking.  (An unterminated opening delimiter
> > gets font-lock-warning-face, a terminated one doesn't.)

> If everything after R"( is fontified as a string, it serves as a 
> "warning" of sort as well.

That means putting syntax-table properties on every " after the R"foo(,
otherwise that string would only extend to the first such ".

But it wouldn't work well, since it wouldn't tell the user whether her
string was already closed.

> > This is the sort of feature which I'm not willing to sacrifice.

> Is it worth a full buffer scan every time you write a new raw string 
> literal?

Yes, definitely.

> >> Then font-lock would fontify the following text as string contents
> >> (until the end of the window or a little bit farther). Then you type
> >> the closer, it only has to scan a little far back (it'll call
> >> syntax-ppss to find the string opener), the closer is propertized as
> >> appropriate, and that's that. No full buffer scans at any step.

> >> I recall that fontifying the rest of the buffer as text after a simple
> >> string opener could be a sore topic for you, but raw strings should be
> >> rare enough (aren't they?), or if they are not, fontification logic
> >> could opt to do something different, while syntax-table properties will
> >> be applied the "correct" way.

> > I'm not sure what you mean by "as text".

> Sorry, I meant "as string".

OK.

Same procedure for a simple string - if it's a terminated string the "
gets font-lock-string-face, if it's not it gets f-l-warning face.

> > I've no reason to think raw strings are at all rare.  I've had
> > several bug reports for them.  I'm not sure what you mean by
> > "fontification logic ... something different" - do you mean in the
> > raw string case?

> I mean that if a raw string is unterminated, the default behavior should 
> be to fontify the rest of the buffer as string. But then again, you 
> could choose some different highliting in font-lock rules.

The current strategy is to fontify the unterminated R"foo( with
warning-face, and let the devil deal with the rest of the string (i.e. no
attempt is made to apply syntax-table properties).  The first portion of
the raw string will indeed get string-face.

As soon as the closing delimiter is typed, the warning-face is removed
from the opener and syntax-table text properties applied throughout the
string.  The entire string then gets string-face.

[ .... ]

> >> Yes, I think before-change-functions should become empty. Or much
> >> emptier.

> > It can't become empty.  after-change-functions is fine for dealing
> > with insertions, but can't do much after a deletion.  Consider the
> > case where you're in a string and all you know is that 5 characters
> > have been deleted.  Those characters might have been )foo", so after
> > checking the beginning of the string starts off with R"bar(, you've
> > then got to scan a long way forward looking for )bar".  Effectively
> > every deletion within a string would involve scanning to the end of
> > that string.

> This is an example of extra complexity you have to retain to implement 
> the above feature the way you do.

It will become more complex and slower, if information from
before-change-functions is ignored, or discarded.  The alternative is,
after each deletion, to scan forward checking that the terminating
delimiter still exists.  This is slower and more complicated than
checking in b-c-f whether it's about to be removed.

> It's probably also an example of how before/after-change-functions 
> essentially duplicate the knowledge of language syntax. I'm guessing 
> here, but to make it work like that, you need to have multiple functions 
> "understand" the raw string syntax.

b/a-c-f implement the language syntax.  It's one of the places the
language is codified.  The mechanism is in several functions, yes.  If
you're interested, go into cc-engine.el and search for "raw string".

> Whereas with syntax-propertize-function, that knowledge is concentrated 
> in one place (maybe two, if font-lock rules do something unusual). This 
> way, the code is simplified.

No, it gets complicated, assuming no loss of functionality.  A given
amount of functionality would get squashed into a smaller place.  The
current implementation (of C++ raw strings) is optimised such that normal
insertion and deletion don't cause the s-t properties on the entire
string to be modified.  That requires details of the buffer before the
insertions and deletions.

[ .... ]

> >>> Maybe, but with a slowdown.  More of these properties will get erased
> >>> than needed (with nested template forms), and they will all need to get
> >>> put back again.

> >> We won't really know until we can measure the result.

> > What's the point in investing all the effort to make the change, when
> > there's not even a prediction of a speed up?

> In principle, the speed-up will come from:

> - Deferred execution (where several buffer changes can be handled 
> together and not right away),

I've never been wholly convinced by laziness.  Sooner or later these
changes need to be handled, and delaying them is not going to accelerate
them.

> - No parsing the buffer contents much farther than the current window, 
> in most cases. Which can speed up the majority of user actions. The 
> exceptions will remain slower, but that is often a good tradeoff.

This will involve loss of functionality, as already noted.  And bugs;
whilst typing in normal text, CC Mode has to search backwards for a safe
place, otherwise context fontification can mess things up.  This is an
area where optimisation would be useful.

> > And I'm not sure where the proof of the syntax-propertize mechanism
> > being helpful is.  Has anybody but its originator positively chosen
> > to use it, whilst being aware of the alternatives?

> The alternatives being reinventing the relevant logic from zero in each 
> major mode? And writing syntax caching logic each time?

Or writing and using a better framework.

The question remains: has anybody other than Stefan M. freely chosen to
use syntax-ppss and syntax-propertize-function, whilst being aware of
their disadvantages and of alternatives?

Remember, that for an extended period of time syntax-ppss didn't work
properly, and even now it doesn't do the right thing in narrowed buffers,
at least for a programming mode such as CC Mode.  The syntax-propertize
mechanism erases s-t p's in a manner not under the control of the major
mode, which means the major mode needs to implement workarounds (which
are liable to be slow).

> > To become usable for CC Mode, it would need to provide something on
> > before-change-functions to complement what's on a-c-f, and it would need
> > to provide some control to the major mode over which syntax-table
> > properties get erased.

> Not something I can comment on.

Hmmm.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>> It can be done reliably, but it would most likely want help from the
>> major mode (e.g. the C syntax tables give the syntax of strings but
>> don't indicate that those strings are not allowed to span multiple
>> lines).
> How about adding support to the syntax table?

I don't know of any language that defines strings this way, so I'd
rather not.

[ Furthermore, I think there are too many subtly different ways to
  (dis)allow multiline strings in programming languages, so handling it
  in something like `syntax-propertize` is preferable at least until we
  have a firm grasp of what is the design space.  ]

It's OK to try and detect the situation and highlight it as an error,
but that's quite different from defining it as "the official syntax".


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

Hi Alan,

On 21.04.2020 00:19, Alan Mackenzie wrote:

>> Is C++ syntax so ambiguous? Can R"( mean something else?
> 
> C++ syntax is ambiguous in several places, but not here.  R"foo(,
> assuming it's not itself inside a literal, means exactly one thing.  But
> you don't know whether it starts an unterminated string unless you seach
> for a closing delimiter.

Then you can put a certain syntax-table value on it either way. Not 
(string-to-syntax "\""), though.

>>> This affects the font locking.  (An unterminated opening delimiter
>>> gets font-lock-warning-face, a terminated one doesn't.)
> 
>> If everything after R"( is fontified as a string, it serves as a
>> "warning" of sort as well.
> 
> That means putting syntax-table properties on every " after the R"foo(,
> otherwise that string would only extend to the first such ".

Nope. You will put (string-to-syntax "|") on the the raw string opener. 
It doesn't match "ordinary" quotes.

> Same procedure for a simple string - if it's a terminated string the "
> gets font-lock-string-face, if it's not it gets f-l-warning face.

As discussed in nearby messages, this should be doable in a generic way 
outside of CC Mode.

>> I mean that if a raw string is unterminated, the default behavior should
>> be to fontify the rest of the buffer as string. But then again, you
>> could choose some different highliting in font-lock rules.
> 
> The current strategy is to fontify the unterminated R"foo( with
> warning-face, and let the devil deal with the rest of the string (i.e. no
> attempt is made to apply syntax-table properties).  The first portion of
> the raw string will indeed get string-face.
> 
> As soon as the closing delimiter is typed, the warning-face is removed
> from the opener and syntax-table text properties applied throughout the
> string.  The entire string then gets string-face.

That would change. But within limits that are acceptable for you, hopefully.

>>> It can't become empty.  after-change-functions is fine for dealing
>>> with insertions, but can't do much after a deletion.  Consider the
>>> case where you're in a string and all you know is that 5 characters
>>> have been deleted.  Those characters might have been )foo", so after
>>> checking the beginning of the string starts off with R"bar(, you've
>>> then got to scan a long way forward looking for )bar".  Effectively
>>> every deletion within a string would involve scanning to the end of
>>> that string.
> 
>> This is an example of extra complexity you have to retain to implement
>> the above feature the way you do.
> 
> It will become more complex and slower, if information from
> before-change-functions is ignored, or discarded.  The alternative is,
> after each deletion, to scan forward checking that the terminating
> delimiter still exists.  This is slower and more complicated than
> checking in b-c-f whether it's about to be removed.

Removal of micro-optimizations will make things slower only when they 
apply to actual bottlenecks. And those can change with the change of the 
approach.

>> It's probably also an example of how before/after-change-functions
>> essentially duplicate the knowledge of language syntax. I'm guessing
>> here, but to make it work like that, you need to have multiple functions
>> "understand" the raw string syntax.
> 
> b/a-c-f implement the language syntax.  It's one of the places the
> language is codified.  The mechanism is in several functions, yes.  If
> you're interested, go into cc-engine.el and search for "raw string".

Your confirmation is good enough for me for now, thank you.

>> Whereas with syntax-propertize-function, that knowledge is concentrated
>> in one place (maybe two, if font-lock rules do something unusual). This
>> way, the code is simplified.
> 
> No, it gets complicated, assuming no loss of functionality.  A given
> amount of functionality would get squashed into a smaller place.  The
> current implementation (of C++ raw strings) is optimised such that normal
> insertion and deletion don't cause the s-t properties on the entire
> string to be modified.  That requires details of the buffer before the
> insertions and deletions.

True. So maybe Emacs will do "extra" work. It still might turn out to be 
faster for most user interactions.

>> In principle, the speed-up will come from:
> 
>> - Deferred execution (where several buffer changes can be handled
>> together and not right away),
> 
> I've never been wholly convinced by laziness.  Sooner or later these
> changes need to be handled, and delaying them is not going to accelerate
> them.

It's the difference between keeping the whole buffer up-to-date and only 
doing that for a limited range of chars.

>> - No parsing the buffer contents much farther than the current window,
>> in most cases. Which can speed up the majority of user actions. The
>> exceptions will remain slower, but that is often a good tradeoff.
> 
> This will involve loss of functionality, as already noted.  And bugs;
> whilst typing in normal text, CC Mode has to search backwards for a safe
> place, otherwise context fontification can mess things up.  This is an
> area where optimisation would be useful.

That's where syntax-ppss comes in.

>>> And I'm not sure where the proof of the syntax-propertize mechanism
>>> being helpful is.  Has anybody but its originator positively chosen
>>> to use it, whilst being aware of the alternatives?
> 
>> The alternatives being reinventing the relevant logic from zero in each
>> major mode? And writing syntax caching logic each time?
> 
> Or writing and using a better framework.

Well, um. You didn't write a generic framework that would help all major 
mode writers yourself. It's a bit too much to expect that from others 
either.

> The question remains: has anybody other than Stefan M. freely chosen to
> use syntax-ppss and syntax-propertize-function, whilst being aware of
> their disadvantages and of alternatives?

I'm not him.

> Remember, that for an extended period of time syntax-ppss didn't work
> properly, and even now it doesn't do the right thing in narrowed buffers,
> at least for a programming mode such as CC Mode.  The syntax-propertize
> mechanism erases s-t p's in a manner not under the control of the major
> mode, which means the major mode needs to implement workarounds (which
> are liable to be slow).

Narrowing is a tricky thing. I'd wager to say that the vast majority of 
our users don't use it, or use it in very specific ways. So any related 
bugs might go unnoticed or take quite some time to be uncovered.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> The question remains: has anybody other than Stefan M. freely chosen to
> use syntax-ppss and syntax-propertize-function, whilst being aware of
> their disadvantages and of alternatives?

Not quite sure I can answer that.

I can point to cperl-mode which used the exact same approach as
`syntax-propertize` (without using `syntax-propertize` because it didn't
exist yet) and I can point to many major modes that use
`syntax-propertize` successfully without my involvement.  Perl is one of
the languages that requires the most intense help from `syntax-table
text properties.

AFAIK CC-mode is the only "alternative" and last I checked there's only
one human on this planet who really knows what it does.

> The syntax-propertize mechanism erases s-t p's in a manner not under
> the control of the major mode, which means the major mode needs to
> implement workarounds (which are liable to be slow).

I have no idea what you're referring to.


        Stefan