A proposal for the future of open-paren-in-column-0-is-defun-start.

A proposal for the future of open-paren-in-column-0-is-defun-start.

[Alan Mackenzie]

Hello, Emacs.

The working of open-paren-in-column-0-is-defun-start is still functional
in emacs-27 and master, providing the newish variable
comment-use-syntax-ppss is set to nil.

When enabled, it still causes problems in, at the very least, CC Mode.
For example, in the following:

    /*
    (
    */
      ^
    point

, (forward-comment -1) spuriously returns nil and doesn't move point.

The root of the problem is that back_comment (in syntax.c) has no means
of distinguishing open parens in comments and strings from other open
parens.  I propose giving the function such a means.

This would be a text property `syntax-open-paren-in-literal' which would
be applied to each and every open paren at column zero inside a literal.
back_comment could easily and cheaply test this text-property.

`syntax-open-paren-in-literal' would be applied throughout the buffer
when it is first loaded into Emacs, and would be kept up to date by
"super before/after-change-functions".  (These functions would be called
at buffer changes regardless of the value of inhibit-modification-hooks.
We would want some way of inhibiting them being called on a change of
text properties, for efficiency's sake).

With this mechanism in place, o-p-i-c-0-i-d-s could safely be left
enabled, for speed's sake, without causing the all too familiar bugs in
CC Mode.

What do people think?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: A proposal for the future of open-paren-in-column-0-is-defun-start.

[Stefan Monnier]

> With this mechanism in place, o-p-i-c-0-i-d-s could safely be left
> enabled, for speed's sake,

FWIW setting `comment-use-syntax-ppss` to nil is likely make things
*slower* in many cases.

This "newish" variable was only introduced so as to have an escape hatch
if it turned out that the new functionality of relying on syntax-ppss
proved problematic.

Given that I haven't seen a single incident reported since that fateful
Dec 12 2017 night, I think we'd be better off removing
`comment-use-syntax-ppss` altogether.


        Stefan

Re: A proposal for the future of open-paren-in-column-0-is-defun-start.

[Alan Mackenzie]

Hello, Stefan.

On Sun, Apr 05, 2020 at 10:33:17 -0400, Stefan Monnier wrote:
> > With this mechanism in place, o-p-i-c-0-i-d-s could safely be left
> > enabled, for speed's sake,

> FWIW setting `comment-use-syntax-ppss` to nil is likely make things
> *slower* in many cases.

Anyhow, the problem with using syntax-ppss here is it involves lots and
lots of scanning over large buffer portions in certain circumstances.
The o-p-i-c-0-i-d-s approach wouldn't.

Also, syntax-ppss doesn't (necessarily) do the right thing on narrowed
buffers, thus potentially leading to errors.  Why don't you make sure
the buffer is widened before using syntax-ppss?

> This "newish" variable was only introduced so as to have an escape hatch
> if it turned out that the new functionality of relying on syntax-ppss
> proved problematic.

Why is the o-p-i-c-0-i-d-s mechanism made dependent on
comment-use-syntax-ppss being nil?

> Given that I haven't seen a single incident reported since that fateful
> Dec 12 2017 night, I think we'd be better off removing
> `comment-use-syntax-ppss` altogether.

Er, Martin reported a delay of 10 seconds when scrolling a buffer with
the mouse wheel.  How does that not cound as an "incident"?

What you mean by your suggestion is to remove
open-paren-in-column-0-is-defun start altogether.

Both Martin and Richard, who have less powerful machines than we do, are
unhappy about this.

Anyway, what exactly happened on 2017-12-12 that qualifies as a fateful
incident?

Also, you haven't commented on my proposal.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: A proposal for the future of open-paren-in-column-0-is-defun-start.

[Stefan Monnier]

>> Given that I haven't seen a single incident reported since that fateful
>> Dec 12 2017 night, I think we'd be better off removing
>> `comment-use-syntax-ppss` altogether.
>
> Er, Martin reported a delay of 10 seconds when scrolling a buffer with
> the mouse wheel.  How does that not cound as an "incident"?

I don't see anything in this report that relates it to
`comment-use-syntax-ppss` (nor to `open-paren-in-column-0-is-defun`).

> What you mean by your suggestion is to remove
> open-paren-in-column-0-is-defun start altogether.

No, my suggestion is to remove `comment-use-syntax-ppss` (i.e. remove
the possibility to set it to nil).

> Both Martin and Richard, who have less powerful machines than we do, are
> unhappy about this.

I don't see any reason to think their unhappiness is due to
`open-paren-in-column-0-is-defun` or `comment-use-syntax-ppss`.

> Anyway, what exactly happened on 2017-12-12 that qualifies as a fateful
> incident?

Commit 14b95587520959c5b54356547a0a69932a9bb480!


        Stefan

Re: A proposal for the future of open-paren-in-column-0-is-defun-start.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > FWIW setting `comment-use-syntax-ppss` to nil is likely make things
  > *slower* in many cases.

In that case, we could arrange for open-paren-in-column-0-is-defun-start
to be heeded _along with_ comment-use-syntax-ppss.

I've seen several arguments which implicitly presume that making
open-paren-in-column-0-is-defun-start work again is impossible or
unthinkable.  That cannot be so.



-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: A proposal for the future of open-paren-in-column-0-is-defun-start.

[Stefan Monnier]

> In that case, we could arrange for open-paren-in-column-0-is-defun-start
> to be heeded _along with_ comment-use-syntax-ppss.

Be my guest.  But before accepting such code, I'll want to see some
concrete evidence that it has a real benefit.

So far, all concrete evidence we have has no direct connection to
open-parens-in-column-0 (other than some people's gut feelings).

We should treat this performance problem the same as any other: do the
profiling, look into the code that takes the major part of the time,
look into why it takes so much time, ...

So far we have: scrolling is too slow in some C buffer and the profile
indicates it happens "somewhere in jit-lock".  There's a lot that might
be going on "in jit-lock", so we've barely scratched the surface of
the problem.

My gut feeling is that we're going to find that we're not spending much
of that time in `back_comment` (the C function which used this
open-paren-in-column-0 heuristic).  This seems to be confirmed by the
fact that setting comment-use-syntax-ppss to nil made no difference: the
problem is elsewhere and we're just wasting time discussing this old
heuristic whose time has passed.


        Stefan

Re: A proposal for the future of open-paren-in-column-0-is-defun-start.

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 06 Apr 2020 11:40:05 -0400
> Cc: acm@muc.de, emacs-devel@gnu.org
> 
> So far we have: scrolling is too slow in some C buffer and the profile
> indicates it happens "somewhere in jit-lock".

Actually, we had a few detailed profiles posted which show many levels
below "somewhere in jit-lock", at least for CC Mode.

Re: A proposal for the future of open-paren-in-column-0-is-defun-start.

[Stefan Monnier]

> Actually, we had a few detailed profiles posted which show many levels
> below "somewhere in jit-lock", at least for CC Mode.

Hmm... I must have missed them.  Sorry 'bout that.
Do we then know where the time is really spent?


        Stefan

Re: A proposal for the future of open-paren-in-column-0-is-defun-start.

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: rms@gnu.org,  acm@muc.de,  emacs-devel@gnu.org
> Date: Mon, 06 Apr 2020 12:25:51 -0400
> 
> > Actually, we had a few detailed profiles posted which show many levels
> > below "somewhere in jit-lock", at least for CC Mode.
> 
> Hmm... I must have missed them.  Sorry 'bout that.
> Do we then know where the time is really spent?

I think the profiles do show that, at least for CC Mode when
fontifying a C buffer (xdisp.c).

I do agree, of course, that we should profile and measure more before
making any firm conclusions.  But let's not dismiss what we already
have.

Re: A proposal for the future of open-paren-in-column-0-is-defun-start.

[Stefan Monnier]

>> > Actually, we had a few detailed profiles posted which show many levels
>> > below "somewhere in jit-lock", at least for CC Mode.
>> Hmm... I must have missed them.  Sorry 'bout that.
>> Do we then know where the time is really spent?
> I think the profiles do show that, at least for CC Mode when
> fontifying a C buffer (xdisp.c).

Sadly, I don't know what they say.  Could someone point me to the
corresponding message(s) (either pointers to the emacs-devel archive or
message-ids would do fine).


        Stefan

Re: A proposal for the future of open-paren-in-column-0-is-defun-start.

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: rms@gnu.org,  acm@muc.de,  emacs-devel@gnu.org
> Date: Mon, 06 Apr 2020 13:49:10 -0400
> 
> > I think the profiles do show that, at least for CC Mode when
> > fontifying a C buffer (xdisp.c).
> 
> Sadly, I don't know what they say.  Could someone point me to the
> corresponding message(s) (either pointers to the emacs-devel archive or
> message-ids would do fine).

Sorry, I didn't realize you were unable to find them.  Here:

  https://lists.gnu.org/archive/html/emacs-devel/2020-04/msg00010.html
  https://lists.gnu.org/archive/html/emacs-devel/2020-04/msg00025.html

Re: A proposal for the future of open-paren-in-column-0-is-defun-start.

[andres.ramirez]

Hi. Stefan.

Stefan> Sadly, I don't know what they say.  Could someone point me to
Stefan> the corresponding message(s) (either pointers to the
Stefan> emacs-devel archive or message-ids would do fine).

https://lists.gnu.org/archive/html/emacs-devel/2020-03/msg00720.html

Best Regards

Re: A proposal for the future of open-paren-in-column-0-is-defun-start.

[martin rudalics]

 > Actually, we had a few detailed profiles posted which show many levels
 > below "somewhere in jit-lock", at least for CC Mode.

When comparing -O0 with -O3 builds I noticed one additional difference
with simple repetitions of 'scroll-up' in xdisp.c.  The optimized build
spent 25% of the CPU time consumed collecting garbage, the debug build
42%.  Does an -O0 build produce more garbage or is it collecting slower?

martin

Re: A proposal for the future of open-paren-in-column-0-is-defun-start.

[Stefan Monnier]

>> Actually, we had a few detailed profiles posted which show many levels
>> below "somewhere in jit-lock", at least for CC Mode.
> When comparing -O0 with -O3 builds I noticed one additional difference
> with simple repetitions of 'scroll-up' in xdisp.c.  The optimized build
> spent 25% of the CPU time consumed collecting garbage, the debug build
> 42%.  Does an -O0 build produce more garbage or is it collecting slower?

I'd guess that it's just the fact that the GC spends almost all its time
manipulating Lisp_Object values, so it is more affected by inlining of
those low-level functions that extract the tag and things like that.


        Stefan

Re: A proposal for the future of open-paren-in-column-0-is-defun-start.

[Eli Zaretskii]

> Cc: acm@muc.de, rms@gnu.org, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Tue, 7 Apr 2020 10:33:58 +0200
> 
>  > Actually, we had a few detailed profiles posted which show many levels
>  > below "somewhere in jit-lock", at least for CC Mode.
> 
> When comparing -O0 with -O3 builds I noticed one additional difference
> with simple repetitions of 'scroll-up' in xdisp.c.  The optimized build
> spent 25% of the CPU time consumed collecting garbage, the debug build
> 42%.  Does an -O0 build produce more garbage or is it collecting slower?

I think the latter.  I don't think --enable-checking causes
significantly more Lisp object to be produced.  GC is very recursive,
so optimizations that put variables in registers make it faster.

Re: A proposal for the future of open-paren-in-column-0-is-defun-start.

[Dmitry Gutov]

Hi Alan,

On 05.04.2020 16:47, Alan Mackenzie wrote:
> `syntax-open-paren-in-literal' would be applied throughout the buffer
> when it is first loaded into Emacs, and would be kept up to date by
> "super before/after-change-functions".  (These functions would be called
> at buffer changes regardless of the value of inhibit-modification-hooks.
> We would want some way of inhibiting them being called on a change of
> text properties, for efficiency's sake).
> 
> With this mechanism in place, o-p-i-c-0-i-d-s could safely be left
> enabled, for speed's sake, without causing the all too familiar bugs in
> CC Mode.
> 
> What do people think?

While I'm not convinced that syntax-ppss is a significant bottleneck 
now, nor crazy about the idea of yet another kind of hooks which will 
have to scan the whole buffer after every buffer change, I say go for 
it. :-)

The proposed approach seems easy enough to implement, and then you could 
do some real-world benchmarks (something like: 1. scrolling 50 times, 2. 
typing new code in a big file, 3. simply visiting a big file), compare 
and see what improved, and what didn't, and by how much. Then we'd have 
something to discuss, at least.

Re: A proposal for the future of open-paren-in-column-0-is-defun-start.

[Stefan Monnier]

> `syntax-open-paren-in-literal' would be applied throughout the buffer
> when it is first loaded into Emacs,

I.e. whole-buffer parsing when opening a file.


        Stefan

Re: A proposal for the future of open-paren-in-column-0-is-defun-start.

[Dmitry Gutov]

On 07.04.2020 00:42, Stefan Monnier wrote:
>> `syntax-open-paren-in-literal' would be applied throughout the buffer
>> when it is first loaded into Emacs,
> I.e. whole-buffer parsing when opening a file.

Which CC Mode does anyway already. ¯\_(ツ)_/¯

Other modes could conceivably apply it in syntax-propertize-function 
instead. 👺 Though I doubt its applicability considering not a lot of 
programming languages routinely have open parens in the first column.

Re: A proposal for the future of open-paren-in-column-0-is-defun-start.

[Stefan Monnier]

> Which CC Mode does anyway already. ¯\_(ツ)_/¯

And so does tree-sitter, yes.

> Other modes could conceivably apply it in syntax-propertize-function
> instead. 👺 Though I doubt its applicability considering not a lot of
> programming languages routinely have open parens in the first column.

Fair enough,


        Stefan