bug#36474: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

bug#36474: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Alan Mackenzie]

Hello João and Emacs.

This is a follow up bug to bug #36423: 27.0.50; electric-pair-mode not
working properly depending of file content.

Start the Emacs master (up to date state as of 2019-07-02T14:30 +0000)
with emacs -Q, put the following in a C++ Mode buffer and enable
electric-pair-mode:

"foo\n

.  Type a " at the end of foo.  electric-pair-mode wrongly inserts two
"s.

Diagnosis: electric-pair--unbalanced-strings-p works after the (single)
newly typed " has been stripped from the buffer.  It attempts to
determine whether there are any open strings after the point of
insertion.  It does this by using parse-partial-sexp, and checks (nth 3
<result>) as evidence of an open string.

This does not work in CC Mode, since although there is an open string
marker (with a string fence syntax-table property on it) this is
"closed" (from parse-partial-sexp's point of view) by the string fence
property on the newline at the end of the line.
electric-pair--unbalanced-strings-p thus returns the wrong result.

A more suitable algorithm might look something like this: check whether
the newly inserted " has a string fence syntax-table text property.
(Its insertion will have already triggered the before- and
after-change-functions which set this property.)  If so, there is an
open string.  Of course, this only applies to CC Mode modes.

-- 
Alan Mackenzie (Nuremberg, Germany).

Fwd: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 3817 bytes --]

On Tue, Jul 2, 2019 at 2:16 PM Alan Mackenzie <acm@muc.de> wrote:

> Hello João and Emacs.

Hello Alan and Emacs,

> This is a follow up bug to bug #36423: 27.0.50; electric-pair-mode not
> working properly depending of file content.

[Did you mean to copy bug-gnu-emacs@gnu.org, or emacs-devel@gnu.org?
I'm assuming the latter and correcting. And sorry for the duplicate reply
Alan.]

> Start the Emacs master (up to date state as of 2019-07-02T14:30 +0000)
> with emacs -Q, put the following in a C++ Mode buffer and enable
> electric-pair-mode:
>
> "foo\n
>
> .  Type a " at the end of foo.  electric-pair-mode wrongly inserts two
> "s.
>
> Diagnosis: electric-pair--unbalanced-strings-p works after the (single)
> newly typed " has been stripped from the buffer.  It attempts to
> determine whether there are any open strings after the point of
> insertion.  It does this by using parse-partial-sexp, and checks (nth 3
> <result>) as evidence of an open string.

I'm afraid this is a (another?) direct consequence of the NL-terminated
strings feature that you introduced more than one year ago.  If you
remember, this had various consequences vis-a-vis balancing,
broke a test (one that I disabled in the expectation that a fix would be
made available, which I don't think happened). Here are some points of
that thread:

https://lists.gnu.org/archive/html/emacs-devel/2018-06/msg00551.html
https://lists.gnu.org/archive/html/emacs-devel/2018-06/msg00580.html

I think I made my views clear back then: NL-terminated strings are a
misfeature. The only argument _for_ them, that they mimic what some
compilers do, is very weak because (1) the code is invalid in both
situations (not in any way "slightly less" invalid in any of them) and
(2) compilers don't edit code and so have different requirements.

The arguments _against_ NL-terminated strings is that they (1) break
longstanding features such as sexp-based navigation (e.g. `up-list`
and friends) for modes such say, `js-mode` and (2) break features
that expect this to work, most notably electric-pair-mode.

Moving forward:

1. We can consider that electric-pair-mode is doing the right thing.
Indeed if NL is indeed terminating a string, then quote balance has been
maintained after the double quote insertion, i.e. it has not worsened.
That is the general contract of  `electric-pair-preserve-balance`.

2.The NL-terminated string feature is removed (or, if you prefer, is
made disableable). This would restore the behaviour that most users
would expect coming over, from say python-mode or js-mode. Perhaps
it can already be disabled with a couple of lines of emacs-lisp tweaking
the syntax-table.

3. Someone comes up with a suitable indirection that doesn't involve
hardcoding `cc-mode` in elec-pair.el.  That indirection would
presumably do what you want for modes `cc-mode` derived
from cc-mode.

4. Someone reinvents electric-pair-mode in cc-model.el.
Let's not do this.

I prefer 2.

Thanks,
João



>


>
> This does not work in CC Mode, since although there is an open string
> marker (with a string fence syntax-table property on it) this is
> "closed" (from parse-partial-sexp's point of view) by the string fence
> property on the newline at the end of the line.
> electric-pair--unbalanced-strings-p thus returns the wrong result.
>
> A more suitable algorithm might look something like this: check whether
> the newly inserted " has a string fence syntax-table text property.
> (Its insertion will have already triggered the before- and
> after-change-functions which set this property.)  If so, there is an
> open string.  Of course, this only applies to CC Mode modes.
>
> --
> Alan Mackenzie (Nuremberg, Germany).



--
João Távora

[-- Attachment #2: Type: text/html, Size: 5293 bytes --]

bug#36474: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Alan Mackenzie]

On Tue, Jul 02, 2019 at 15:13:35 +0100, João Távora wrote:
>  > Hello João and Emacs.

> Hello Alan and Emacs,

> On Tue, Jul 2, 2019 at 2:16 PM Alan Mackenzie <acm@muc.de> wrote:

> > This is a follow up bug to bug #36423: 27.0.50; electric-pair-mode not
> > working properly depending of file content.

> [did you mean to copy bug-gnu-emacs@gnu.org, or emacs-devel@gnu.org?
> I'm assuming the latter and correcting]

No, it's a bug, therefore I submitted a bug report.

> > Start the Emacs master (up to date state as of 2019-07-02T14:30 +0000)
> > with emacs -Q, put the following in a C++ Mode buffer and enable
> > electric-pair-mode:

> > "foo\n

> > .  Type a " at the end of foo.  electric-pair-mode wrongly inserts two
> > "s.

> > Diagnosis: electric-pair--unbalanced-strings-p works after the (single)
> > newly typed " has been stripped from the buffer.  It attempts to
> > determine whether there are any open strings after the point of
> > insertion.  It does this by using parse-partial-sexp, and checks (nth 3
> > <result>) as evidence of an open string.

> I'm afraid this is a (another?) direct consequence of the NL-terminated
> strings feature that you introduced more than one year ago.  If you
> remember, this had various consequences vis-a-vis balancing,
> broke a test (one that I disabled in the expectation that a fix would be
> made available, which I don't think happened). Here are some points of
> that thread:

> https://lists.gnu.org/archive/html/emacs-devel/2018-06/msg00551.html
> https://lists.gnu.org/archive/html/emacs-devel/2018-06/msg00580.html

> I think I made my views clear back then: NL-terminated strings are a
> misfeature. The only argument _for) them, that they mimic what some
> compilers do, is very weak because (1) the code is invalid in both
> situations (not in any way "slightly less" invalid in any of them) and
> (2) compilers don't edit code and so have different requirements.

> The arguments _against_ NL-terminated strings is that they (1) break
> longstanding features such as sexp-based navigation (e.g. `up-list`
> and friends) for modes such say, `js-mode` and (2) break features
> that expect this to work, most notably electric-pair-mode.

This isn't true.  If those other feature no longer work with an up to
date Emacs, they should be fixed.

The fontification that CC Mode does is natural and helpful, and users
haven't complained about it (except when there've been bugs).  There have
certainly been no complaints about using font-lock-warning-face for the
invalid string delimiters, and font-lock-string-face for valid ones.

> Moving forward:

> 1. We can consider that electric-pair-mode is doing the right thing.
> Indeed if NL is indeed terminating a string, then quote balance has been
> maintained after the double quote insertion, i.e. it has not worsened.
> That is the general contract of  `electric-pair-preserve-balance`.

There is a bug: on typing a " to close a string, two "s are inserted into
the buffer, the second one being invalid.  This make absolutely no sense
from a user point of view.

> 2.The NL-terminated string feature is removed (or, if you prefer, is
> made disableable). This would restore the behaviour that most users
> would expect coming over, from say python-mode or js-mode. Perhaps
> it can already be disabled with a couple of lines of emacs-lisp tweaking
> the syntax-table.

The invalid string feature is here to stay.  It is a positive user
feature.  CC Mode has often been a pioneer in inventing Emacs features,
and this is just such a feature.

> 3. Someone comes up with a suitable indirection that doesn't involve
> hardcoding `cc-mode` in elec-pair.el.  That indirection would
> presumably do what you want for modes `cc-mode` derived
> from cc-mode.

There is already a great deal of such indirection in electric-pair-mode.
(Look for "(funcall electric-pair-....)" in elec-pair.el.)  Maybe there
is enough there already to accomodate CC Mode, maybe an extra function
variable would need introducing.

For this, I think we would both rather that you amend elec-pair.el rather
than me.

> 4. Someone reinvents electric-pair-mode in cc-model.el.
> Let's not do this.

No, let's not do that!  :-)

> I prefer 2.

That isn't an option.  Unless you can come up with another workable
strategy that achieves the same effect.

> Thanks,
> João



> > This does not work in CC Mode, since although there is an open string
> > marker (with a string fence syntax-table property on it) this is
> > "closed" (from parse-partial-sexp's point of view) by the string fence
> > property on the newline at the end of the line.
> > electric-pair--unbalanced-strings-p thus returns the wrong result.

> > A more suitable algorithm might look something like this: check whether
> > the newly inserted " has a string fence syntax-table text property.
> > (Its insertion will have already triggered the before- and
> > after-change-functions which set this property.)  If so, there is an
> > open string.  Of course, this only applies to CC Mode modes.

> --
> João Távora

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 3741 bytes --]

On Tue, Jul 2, 2019 at 5:04 PM Alan Mackenzie <acm@muc.de> wrote:

> > [did you mean to copy bug-gnu-emacs@gnu.org, or emacs-devel@gnu.org?
> > I'm assuming the latter and correcting]
> No, it's a bug, therefore I submitted a bug report.

You should use the X-Debbugs-CC feature then (and why not continue
in the existing bug 36423?)

Anyway, I insist this matter be brought to emacs-devel because it's a
followup to a discussion that started there but never reached a
suitable conclusion. For that reason, and because I provide a
workaround for the bug  at the end of  this message, I'm cross-posting
this one mail to both the bug list and emacs-devel.

> > The arguments _against_ NL-terminated strings is that they (1) break
> > longstanding features such as sexp-based navigation (e.g. `up-list`
> > and friends) for modes such say, `js-mode` and (2) break features
> > that expect this to work, most notably electric-pair-mode.
>
> This isn't true.

What "isn't true"? Have those features broken or not? They worked
before the fe06f643b commit and don't work after the commit. It
sounds quite unrefutable to me.

> If those other feature no longer work with an up to
> date Emacs, they should be fixed.

I've stated this repeatedly in the life of this discussion: it's not just
about
electric-pair-mode. If you try to M-x up-list from a multi-line string in
emacs 26.1 it works just as well in js-mode and c++-mode.  In emacs
master it does not in c++-mode. Same with forward-sexp on the starting
delimiter, etc.

> The fontification that CC Mode does is natural and helpful, and users
> haven't complained about it (except when there've been bugs).

Yes, users haven't complained except when users have complained.

> There have
> certainly been no complaints about using font-lock-warning-face for the
> invalid string delimiters, and font-lock-string-face for valid ones.

That's because providing this annotation is perfectly fine.  The problem
is providing it _at the expense of other features_. And _that's_ what
they've complained about: an average user has no obvious way of
telling that the particular implementation of the red annotation thingy
is guilty of breaking his C-M-u or his electric-pair-mode.

He/she might even judge the latter more vital than said red thingy
, an annotation which he/she will get by other means if using
very popular packages such as flycheck, or flymake, or eglot, or
lsp-mode, etc. These normally call the compiler directly on the
source code and highlight those and many other errors.

On the other hand, if what you want is the red annotation, are you
absolutely sure there isn't a better way to get it? And if you are,
are you also absolutely sure you need to put it in the code and
and not provide an easy way to turn it off?

> For this, I think we would both rather that you amend elec-pair.el rather
> than me.

I'll be "mulling this over". There are potentially many other points of
breakage that would need such an indirection, and doing that to serve
just a particular cc-mode quirk doesn't sound priority to me.

In the meantime, let others chime in.

Also, in the meantime, for a user that is bothered by this bug,
I'd recomend to put something like this in his/her .emacs file:

  (defun c-unescaped-nls-in-string-p (&optional quote-pos) t)

I had something more elaborate in my setup but just this
seems to fix it in my testing.

There is a also a very promising variable, c-multiline-string-start-char,
that I think would be a good candidate for customizing this, but I
haven't messed with it enough. Just setting it from .emacs doesn't
do the trick. Perhaps in a mode hook?

--
João Távora

[-- Attachment #2: Type: text/html, Size: 5107 bytes --]

bug#36474: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Alan Mackenzie]

Hello, João.

On Tue, Jul 02, 2019 at 18:22:34 +0100, João Távora wrote:
> On Tue, Jul 2, 2019 at 5:04 PM Alan Mackenzie <acm@muc.de> wrote:

> > > [did you mean to copy bug-gnu-emacs@gnu.org, or emacs-devel@gnu.org?
> > > I'm assuming the latter and correcting]
> > No, it's a bug, therefore I submitted a bug report.

> You should use the X-Debbugs-CC feature then (and why not continue
> in the existing bug 36423?)

I didn't know about the former, and as for the bug, it is a different
bug with differenct causes from #36423.

> Anyway, I insist this matter be brought to emacs-devel because it's a
> followup to a discussion that started there but never reached a
> suitable conclusion. For that reason, and because I provide a
> workaround for the bug  at the end of  this message, I'm cross-posting
> this one mail to both the bug list and emacs-devel.

> > > The arguments _against_ NL-terminated strings is that they (1) break
> > > longstanding features such as sexp-based navigation (e.g. `up-list`
> > > and friends) for modes such say, `js-mode` and (2) break features
> > > that expect this to work, most notably electric-pair-mode.

> > This isn't true.

> What "isn't true"? Have those features broken or not?

They may well be broken.  CC Mode hasn't broken them.  They made invalid
assumptions, which turned out to be unjustified.

> They worked before the fe06f643b commit and don't work after the
> commit. It sounds quite unrefutable to me.

I don't know what you're talking about.

> > If those other feature no longer work with an up to date Emacs, they
> > should be fixed.

> I've stated this repeatedly in the life of this discussion: it's not
> just about electric-pair-mode. If you try to M-x up-list from a
> multi-line string in emacs 26.1 it works just as well in js-mode and
> c++-mode.  In emacs master it does not in c++-mode. Same with
> forward-sexp on the starting delimiter, etc.

I've just tried these in a multiline string in C++ Mode.  Both up-list
and forward-sexp work just fine.  I don't know what you're doing.

> > The fontification that CC Mode does is natural and helpful, and users
> > haven't complained about it (except when there've been bugs).

> Yes, users haven't complained except when users have complained.

> > There have certainly been no complaints about using
> > font-lock-warning-face for the invalid string delimiters, and
> > font-lock-string-face for valid ones.

> That's because providing this annotation is perfectly fine.  The problem
> is providing it _at the expense of other features_.

If other features are broken (and your list of other broken features, so
far, is empty), they should be fixed.

> And _that's_ what they've complained about: an average user has no
> obvious way of telling that the particular implementation of the red
> annotation thingy is guilty of breaking his C-M-u or his
> electric-pair-mode.

That's groundless disparagement.  C-M-u works, and electric-pair-mode is
broken because it's broken.  In one place it's using scan-sexps to move
forward over whitespace, totally oblivious to the possibility of
syntax-table text properties (which have been in use since Emacs-21).
That's broken code.

> He/she might even judge the latter more vital than said red thingy
> , an annotation which he/she will get by other means if using
> very popular packages such as flycheck, or flymake, or eglot, or
> lsp-mode, etc. These normally call the compiler directly on the
> source code and highlight those and many other errors.

Irrelevant.

> On the other hand, if what you want is the red annotation, are you
> absolutely sure there isn't a better way to get it?

No, I'm not.  That's why I invited you to come up with a better way, if
you can.

> And if you are, are you also absolutely sure you need to put it in the
> code and and not provide an easy way to turn it off?

It's a core feature of the mode, not an option.

> > For this, I think we would both rather that you amend elec-pair.el rather
> > than me.

> I'll be "mulling this over". There are potentially many other points
> of breakage

"Potentially" many?  So far, there is precisely one, in
electric-pair--unbalanced-strings-p.  I thought I was doing you a favour
by diagnosing the trouble.  If I'd known I'd get the reaction from you
I've just got, I wouldn't have bothered.

> that would need such an indirection, and doing that to serve just a
> particular cc-mode quirk doesn't sound priority to me.

No, you'd be cleaning up your code, to conform with the reality that in
2019 major modes use syntax-table text properties.  Features from CC
Mode have a habit of migrating to the Emacs core.

> In the meantime, let others chime in.

> Also, in the meantime, for a user that is bothered by this bug,
> I'd recomend to put something like this in his/her .emacs file:

>   (defun c-unescaped-nls-in-string-p (&optional quote-pos) t)

It's free software, but that's a stupid thing to do.

> I had something more elaborate in my setup but just this
> seems to fix it in my testing.

> There is a also a very promising variable, c-multiline-string-start-char,
> that I think would be a good candidate for customizing this, ....

It is not a customisation variable.  It is a language definition
variable.

> .... but I haven't messed with it enough. Just setting it from .emacs
> doesn't do the trick. Perhaps in a mode hook?

Or, alternatively, actually fix the problems which have been festering
for years or decades, and are just now revealing themselves.  Thus far,
there's exactly one such problem in electric-pair--unbalanced-strings-p.

> --
> João Távora

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: bug#36474: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Stefan Monnier]

>> And if you are, are you also absolutely sure you need to put it in the
>> code and and not provide an easy way to turn it off?
> It's a core feature of the mode, not an option.

I don't understand this statement.
How could handling of incorrect code be part of "the core feature" of
the mode?  I understand you consider this to be a useful feature, and
even though I don't find it of value I'm sure some other users may like
it, but .... "core feature"?


        Stefan

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 3533 bytes --]

On Tue, Jul 2, 2019 at 7:28 PM Alan Mackenzie <acm@muc.de> wrote:
> Hello, João.

> I didn't know about the former, and as for the bug, it is a different
> bug with differenct causes from #36423.

If you say so... I stopped cross-posting, you probably should, too.

> > > This isn't true.
>
> > What "isn't true"? Have those features broken or not?
>
> They may well be broken.  CC Mode hasn't broken them.  They made invalid
> assumptions, which turned out to be unjustified.

C-M-u (up-list) made an invalid assumption?

> > They worked before the fe06f643b commit and don't work after the
> > commit. It sounds quite unrefutable to me.
>
> I don't know what you're talking about.
> I've just tried these in a multiline string in C++ Mode.  Both up-list
> and forward-sexp work just fine.  I don't know what you're doing.

Start emacs -Q, open a buffer with a double quote followed by some
newlines and then another another double quote.  Now put the buffer in
C++ mode and type C-M-u from between the quotes.  Instead of up-listing
to the first quote, you get an error.  Now but point on the starting
quote and try C-M-f.  Again, error. But only in Emacs 27/cc-mode,
probably starting from fe06f64b (that's a git commit hash).

Now repeat this experiment. Let me make a little ASCII table:

                     WORKS    DOESN'T

emacs 27/c++mode                X
emacs 26.1/c++-mode   X
emacs 27/js-mode      X
emacs 27/ruby-mode    X
emacs 27/(many)       X

Do you know what I'm talking about now?

Now, you might not care for this consistency, but I personally did.

I've always had my methods of navigating code with unterminated strings,
with _or without_ electric-pair mode and your cc-mode changes broke
this workflow.

> > lsp-mode, etc. These normally call the compiler directly on the
> > source code and highlight those and many other errors.
>
> Irrelevant.
>
> > On the other hand, if what you want is the red annotation, are you
> > absolutely sure there isn't a better way to get it?
>
> No, I'm not.  That's why I invited you to come up with a better way, if
> you can.

I gave your one exactly one paragraph ago, which you called dismissed as
"irrelevant". *shrug*.

> > I'll be "mulling this over". There are potentially many other points
> > of breakage
> "Potentially" many?  So far, there is precisely one, in
> electric-pair--unbalanced-strings-p.

What about d37d30cef5bbbdf8d17315835126d76d4681b22a? Is that the same
problem?  Maybe, I don't know.  Certainly the broken C-M-u and C-M-f,
aren't the same problem.

> No, you'd be cleaning up your code, to conform with the reality that in
> 2019 major modes use syntax-table text properties.  Features from CC
> Mode have a habit of migrating to the Emacs core.

It's not "my" code and I won't be bullied into making changes I don't
agree with.  Things worked fine until you added a feature in June 2018
that broke them.  You refuse to even let the user disable that feature
or consider alternatives.  That's simply not reasonable to me.

> > Also, in the meantime, for a user that is bothered by this bug,
> > I'd recomend to put something like this in his/her .emacs file:
>
> >   (defun c-unescaped-nls-in-string-p (&optional quote-pos) t)
>
> It's free software, but that's a stupid thing to do.

I'd ask you to actually give an argument instead of an insult, but I've
been on this hamster wheel before, so never mind.  It works quite well.

--
João Távora

[-- Attachment #2: Type: text/html, Size: 4154 bytes --]

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Alan Mackenzie]

Hello, João.

On Wed, Jul 03, 2019 at 10:28:11 +0100, João Távora wrote:
> On Tue, Jul 2, 2019 at 7:28 PM Alan Mackenzie <acm@muc.de> wrote:

> > I didn't know about the former, and as for the bug, it is a different
> > bug with differenct causes from #36423.

> If you say so... I stopped cross-posting, you probably should, too.

You've hijacked this bug report for your own purposes.  That is not a
gentlemanly thing to do.  Why could you not have started your own
thread?

Are you going to attend to bug #36474, which was the proper topic of
this thread?  Or would you like me to?

> > > > This isn't true.

> > > What "isn't true"? Have those features broken or not?

> > They may well be broken.  CC Mode hasn't broken them.  They made
> > invalid assumptions, which turned out to be unjustified.

> C-M-u (up-list) made an invalid assumption?

No.  You made an invalid assumption in trying to use it.

> > > They worked before the fe06f643b commit and don't work after the
> > > commit. It sounds quite unrefutable to me.

> > I don't know what you're talking about.
> > I've just tried these in a multiline string in C++ Mode.  Both up-list
> > and forward-sexp work just fine.  I don't know what you're doing.

> Start emacs -Q, open a buffer with a double quote followed by some
> newlines and then another another double quote.  Now put the buffer in
> C++ mode and type C-M-u from between the quotes.

Ah, I see.  You said you'd done this from a multiline string.  What you
have constructed is not a multiline string, it is two invalid string
delimiters unconnected with eachother with extraneous matter between
them.

> Instead of up-listing to the first quote, you get an error.  Now put
> point on the starting quote and try C-M-f.  Again, error.

It works for me.  It moves to the end of the initial invalid string.

> But only in Emacs 27/cc-mode, probably starting from fe06f64b (that's
> a git commit hash).

I'm fully aware it's a git commit hash.  Funnily enough, I don't
memorise such hashes, so if you wish to draw my attention to some
commit, please have the decency to give me its date, and the summary
line of its commit message, instead of expecting me to do your work to
make your point.

> Now repeat this experiment. Let me make a little ASCII table:

>                      WORKS    DOESN'T

> emacs 27/c++mode                X
> emacs 26.1/c++-mode   X
> emacs 27/js-mode      X
> emacs 27/ruby-mode    X
> emacs 27/(many)       X

> Do you know what I'm talking about now?

Yes, I do.  Now try the following.  In your C++ buffer, make a comment,
and put an open parenthesis inside it.  After the comment put a matching
close paren.  From the opening paren try C-M-f.  Guess what?  You get an
error, just like you do in your case with the broken strings.

> Now, you might not care for this consistency, but I personally did.

CC Mode has always put a strong emphasis on correctness and accuracy.
You're complaining that a dubious trick borne of incorrectness no longer
works.  I don't think you have a strong position to complain.

> I've always had my methods of navigating code with unterminated strings,
> with _or without_ electric-pair mode and your cc-mode changes broke
> this workflow.

I think, perhaps, you encounter such broken strings extremely rarely,
and you're making a big song and dance over a relatively minor matter.

> > > lsp-mode, etc. These normally call the compiler directly on the
> > > source code and highlight those and many other errors.

> > Irrelevant.

> > > On the other hand, if what you want is the red annotation, are you
> > > absolutely sure there isn't a better way to get it?

> > No, I'm not.  That's why I invited you to come up with a better way, if
> > you can.

> I gave your one exactly one paragraph ago, which you called dismissed as
> "irrelevant". *shrug*.

*shrug* as much as you like.  You know full well what I meant when I
asked you to suggest an alternative.  So please stop being so damned
evasive and answer the point: can you suggest some other mechanism by
which CC Mode can create the required font-locking?  If you can, and
it's workable, we all win.

> > > I'll be "mulling this over". There are potentially many other points
> > > of breakage
> > "Potentially" many?  So far, there is precisely one, in
> > electric-pair--unbalanced-strings-p.

> What about d37d30cef5bbbdf8d17315835126d76d4681b22a? Is that the same
> problem?  Maybe, I don't know.  Certainly the broken C-M-u and C-M-f,
> aren't the same problem.

C-M-u and C-M-f aren't broken.  You just have unreasonable expectations
of them.

> > No, you'd be cleaning up your code, to conform with the reality that in
> > 2019 major modes use syntax-table text properties.  Features from CC
> > Mode have a habit of migrating to the Emacs core.

> It's not "my" code and I won't be bullied into making changes I don't
> agree with.

If anybody's the bully around here, it's not me.  I found a minor bug in
electric-pair-mode, diagnosed it, and reported it.  Then I found myself
engulfed in your immoderate tirade, trying to bully me into a major
restructuring of CC Mode so as to suit your somewhat unreasonable
wishes.

As far as I can make out, you are the maintainer of electric-pair-mode,
and you seem to be refusing to fix a bug, not even a difficult bug.  Are
you going to fix bug #36474 or not?

> Things worked fine until you added a feature in June 2018 that broke
> them.  You refuse to even let the user disable that feature or
> consider alternatives.  That's simply not reasonable to me.

Like you declining even to look at bug #36474?  Correctness in CC Mode
is not an optional feature - it goes to the heart of the mode.

> > > Also, in the meantime, for a user that is bothered by this bug,
> > > I'd recomend to put something like this in his/her .emacs file:

> > >   (defun c-unescaped-nls-in-string-p (&optional quote-pos) t)

> > It's free software, but that's a stupid thing to do.

> I'd ask you to actually give an argument instead of an insult, but I've
> been on this hamster wheel before, so never mind.  It works quite well.

Making random changes to the innards of a program tends to break things.
Will that do you instead?

> --
> João Távora

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Dmitry Gutov]

On 03.07.2019 13:58, Alan Mackenzie wrote:
>> But only in Emacs 27/cc-mode, probably starting from fe06f64b (that's
>> a git commit hash).
> I'm fully aware it's a git commit hash.  Funnily enough, I don't
> memorise such hashes, so if you wish to draw my attention to some
> commit, please have the decency to give me its date, and the summary
> line of its commit message, instead of expecting me to do your work to
> make your point.

The rest of the discussion aside, IMO it's a very strange reply. Though 
I might be biased, given that a lot of my software-related discussions 
are on platforms that linkify commit hashes.

Even so, is calling 'git show fe06f64b' too hard to do?

I would print its output on this occasion here, but in this case Git 
says it's 'ambiguous'. So maybe Joao should elaborate.

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Alan Mackenzie]

Hello, Dmitry.

On Wed, Jul 03, 2019 at 16:07:59 +0300, Dmitry Gutov wrote:
> On 03.07.2019 13:58, Alan Mackenzie wrote:
> >> But only in Emacs 27/cc-mode, probably starting from fe06f64b (that's
> >> a git commit hash).
> > I'm fully aware it's a git commit hash.  Funnily enough, I don't
> > memorise such hashes, so if you wish to draw my attention to some
> > commit, please have the decency to give me its date, and the summary
> > line of its commit message, instead of expecting me to do your work to
> > make your point.

> The rest of the discussion aside, IMO it's a very strange reply. Though 
> I might be biased, given that a lot of my software-related discussions 
> are on platforms that linkify commit hashes.

> Even so, is calling 'git show fe06f64b' too hard to do?

Yes.  As you'll have noticed, this is an adversarial debate rather than
a constructive cooperation, and expecting me to put in unnecessary work
in order to assist my opponent in making a point against me is not
reasonable.

Surely it is a matter of plain ettiquette: for the writer to put in that
little extra bit of work himself, rather than forcing his readers to do
it.  When writing, I always do this myself.

Besides, other people will read this post, and they'll have to remain
mystified, or put in the work themselves.  Giving a cryptic, rather than
a meaningful, reference thus costs more time than it saves.

Besides[2], the next time we move to a new VCS, the reference will
become totally meaningless.

> I would print its output on this occasion here, but in this case Git 
> says it's 'ambiguous'. So maybe Joao should elaborate.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Dmitry Gutov]

On 03.07.2019 16:32, Alan Mackenzie wrote:

> Yes.  As you'll have noticed, this is an adversarial debate rather than
> a constructive cooperation, and expecting me to put in unnecessary work
> in order to assist my opponent in making a point against me is not
> reasonable.

IME even the adversarial debates, when treated as such, tend to waste 
extra time from all parties involved. You might remember a few of those.

> Surely it is a matter of plain ettiquette: for the writer to put in that
> little extra bit of work himself, rather than forcing his readers to do
> it.  When writing, I always do this myself.

I don't think so. It seems much easier to call 'git show' than to 
manually format a reference in the format that you requested. Some email 
clients might even do that automatically.

I wouldn't want to see this written into the general rules of 
discussions on this mailing list.

> Besides[2], the next time we move to a new VCS, the reference will
> become totally meaningless.

I'm not buying this, sorry. Anybody doing software archaeology in the 
year 2040 would probably be able to dig out a Git repository of Emacs as 
well.

And anyway, it seems like Git is here to stay. By the time we migrate 
off it, it's most likely we've moved away from mailing lists as the 
discussion medium.

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Eli Zaretskii]

> Date: Wed, 3 Jul 2019 13:32:17 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: João Távora <joaotavora@gmail.com>,
>  emacs-devel <emacs-devel@gnu.org>
> 
> > Even so, is calling 'git show fe06f64b' too hard to do?
> 
> Yes.  As you'll have noticed, this is an adversarial debate rather than
> a constructive cooperation

It is only an adversarial debate if you two make it so.  I suggest to
respond as if you don't perceive the adversarial aspects, and expect
the other side to do the same.

> Surely it is a matter of plain ettiquette

It is not.  Showing Git SHA is perfectly okay, at least in this list.

> Besides, other people will read this post, and they'll have to remain
> mystified, or put in the work themselves.  Giving a cryptic, rather than
> a meaningful, reference thus costs more time than it saves.

That ship has sailed when we switched to Git.  It is not wise, to say
the least, to impregnate this discussion, which is already loaded with
emotions, with yet another emotional reaction against something that
no one can do anything about.

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Alan Mackenzie]

Hello, Eli.

On Wed, Jul 03, 2019 at 18:58:39 +0300, Eli Zaretskii wrote:
> > Date: Wed, 3 Jul 2019 13:32:17 +0000
> > From: Alan Mackenzie <acm@muc.de>
> > Cc: João Távora <joaotavora@gmail.com>,
> >  emacs-devel <emacs-devel@gnu.org>

> > > Even so, is calling 'git show fe06f64b' too hard to do?

> > Yes.  As you'll have noticed, this is an adversarial debate rather than
> > a constructive cooperation

> It is only an adversarial debate if you two make it so.  I suggest to
> respond as if you don't perceive the adversarial aspects, and expect
> the other side to do the same.

I've been trying that.  It's not been working well.

> > Surely it is a matter of plain ettiquette

> It is not.  Showing Git SHA is perfectly okay, at least in this list.

> > Besides, other people will read this post, and they'll have to remain
> > mystified, or put in the work themselves.  Giving a cryptic, rather than
> > a meaningful, reference thus costs more time than it saves.

> That ship has sailed when we switched to Git.  It is not wise, to say
> the least, to impregnate this discussion, which is already loaded with
> emotions, with yet another emotional reaction against something that
> no one can do anything about.

That is not true.  A policy decision could be taken now to apply to
future commit references.  I can understand you not wanting to lay down
such a policy, however.

I will continue to supply information such as date and commit message
whenever I refer to a git commit, out of courtesy  and consideration for
the people who will read my post.  They will then immediately know what
I'm talking about, and have enough details to decide whether to look up
the commit in git.

Conversely, I will feel free to ignore a bare git commit hash, for
example when I'm tired, or feeling harrassed, or the tone of the post
containing it is unfriendly, or, quite bluntly, when I just can't be
bothered.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Eli Zaretskii]

> Date: Wed, 3 Jul 2019 20:54:57 +0000
> Cc: dgutov@yandex.ru, joaotavora@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > Yes.  As you'll have noticed, this is an adversarial debate rather than
> > > a constructive cooperation
> 
> > It is only an adversarial debate if you two make it so.  I suggest to
> > respond as if you don't perceive the adversarial aspects, and expect
> > the other side to do the same.
> 
> I've been trying that.  It's not been working well.

Nevertheless, I urge you to keep doing that.

> > > Besides, other people will read this post, and they'll have to remain
> > > mystified, or put in the work themselves.  Giving a cryptic, rather than
> > > a meaningful, reference thus costs more time than it saves.
> 
> > That ship has sailed when we switched to Git.  It is not wise, to say
> > the least, to impregnate this discussion, which is already loaded with
> > emotions, with yet another emotional reaction against something that
> > no one can do anything about.
> 
> That is not true.  A policy decision could be taken now to apply to
> future commit references.  I can understand you not wanting to lay down
> such a policy, however.

We actually have such a policy, but it isn't working, and I don't
think it will ever do.

> Conversely, I will feel free to ignore a bare git commit hash, for
> example when I'm tired, or feeling harrassed, or the tone of the post
> containing it is unfriendly, or, quite bluntly, when I just can't be
> bothered.

I advise against that.

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Yes.  As you'll have noticed, this is an adversarial debate rather than
  > a constructive cooperation,

If that's really the case, how can we change the discussion into
constructive cooperation?  The purpose of this discussion is to
improve Emacs.  When we disagree about how to do that, can we
keep in mind that we are all on the same side?

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 3 Jul 2019 16:07:59 +0300
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
> On 03.07.2019 13:58, Alan Mackenzie wrote:
> >> But only in Emacs 27/cc-mode, probably starting from fe06f64b (that's
> >> a git commit hash).
> > I'm fully aware it's a git commit hash.  Funnily enough, I don't
> > memorise such hashes, so if you wish to draw my attention to some
> > commit, please have the decency to give me its date, and the summary
> > line of its commit message, instead of expecting me to do your work to
> > make your point.
> 
> The rest of the discussion aside, IMO it's a very strange reply.

I agree.  Calling a reference to a commit SHA "indecent" (not in so
many words, but it's clearly hinted upon) is harsh and unfair, because
I cannot imagine any ill will behind such a reference.  Let's please
not use such harsh talk in our discussions.

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1957 bytes --]

On Wed, Jul 3, 2019 at 2:08 PM Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 03.07.2019 13:58, Alan Mackenzie wrote:
> >> But only in Emacs 27/cc-mode, probably starting from fe06f64b (that's
> >> a git commit hash).
> > I'm fully aware it's a git commit hash.  Funnily enough, I don't
> > memorise such hashes, so if you wish to draw my attention to some
> > commit, please have the decency to give me its date, and the summary
> > line of its commit message, instead of expecting me to do your work to
> > make your point.
>
> The rest of the discussion aside, IMO it's a very strange reply. Though
> I might be biased, given that a lot of my software-related discussions
> are on platforms that linkify commit hashes.
>
> Even so, is calling 'git show fe06f64b' too hard to do?
>
> I would print its output on this occasion here, but in this case Git
> says it's 'ambiguous'. So maybe Joao should elaborate.
>

I'm sorry it's fe06f643b2808b198bb58bda04a8c863e55a2a56, the
3 got lost in translation.

BTW I believe this is the commit that broke things, but it might have been
around this time.

commit fe06f643b2808b198bb58bda04a8c863e55a2a56
Author: Alan Mackenzie <acm@muc.de>
Date:   Fri Jun 8 16:42:18 2018 +0000

    CC Mode: Fontify unbalanced quotes in unconstrained multiline strings,
etc.

    ("Unconstrained" meaning that every string is multiline, without
needing such
    special marking as used by Pike Mode.)

    * lisp/progmodes/cc-mode.el (c-pps-to-string-delim): Don't process the
char
    before BOB.
    (c-multiline-string-check-final-quote): New function.
    (c-bc-changed-stringiness): New variable.
    (c-before-change-check-unbalanced-strings): Add handling for
unconstrained
    multiline strings.
    (c-after-change-re-mark-unbalanced-strings): Add handling for
unconstrained
    multiline strings.  Handle escaped double quotes more accurately.





-- 
João Távora

[-- Attachment #2: Type: text/html, Size: 2648 bytes --]

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 3068 bytes --]

On Wed, Jul 3, 2019 at 11:58 AM Alan Mackenzie <acm@muc.de> wrote:

> I'm fully aware it's a git commit hash.  Funnily enough, I don't
> memorise such hashes, so if you wish to draw my attention to some
> commit, please have the decency to give me its date,

Look I'm not being indecent, Alan. Stop this. just `git show fe06f64b`,
and move on.

> You're complaining that a dubious trick borne of incorrectness no longer
> works.

Why are you dictating what I can and can't use that to select
the region between the two quotes? I write my text, fill the pargaph,
select it with C-M-u C-M-SPC (nicely chained "u" and SPC) and then
I C-S-% to put in the backslahes. You may well consider it a "trick"
and "dubious" and "stupid", but it's the way I work.

Anyway, tell me, out of curiosity, how do you do it? How do you create

"this \
text, \
for \
example"

?

> I think, perhaps, you encounter such broken strings extremely rarely,
> and you're making a big song and dance over a relatively minor matter.

No, it's pretty routine, really.  Help blurbs in c++ and multi-line shell
commands in Makefiles, from the top of my head.  C-M-stuff is so
second nature in Emacs I don't even think when I'm using it, I just
expect it to work. I have user commands that use the underlying
functions, too.  By the way C-M-u is backward-up-list, not up-list
as I stated earlier (but both are broken, I think).

> can you suggest some other mechanism by
> which CC Mode can create the required font-locking?  If you can, and
> it's workable, we all win.

No, I can't, but I've asked others to chime in. I would personally do it
(in fact I already do do it) via flymake-mode. If you don't like that,
sorry.

> C-M-u and C-M-f aren't broken.  You just have unreasonable expectations
> of them.

If you call unreasonable expectations gained from many years of using
Emacs in a variety of modes, including, up to very recently, cc-mode,
then clearly there's little point in proceeding this argument.

> > > No, you'd be cleaning up your code, to conform with the reality that
in
> > > 2019 major modes use syntax-table text properties.  Features from CC
> > > Mode have a habit of migrating to the Emacs core.
> > It's not "my" code and I won't be bullied into making changes I don't
> > agree with.
> If anybody's the bully around here, it's not me.  I found a minor bug in
> electric-pair-mode, diagnosed it, and reported it.

You believe this is a bug in e-p-m. It's certainly your right to believe
that. I believe it's a bug in cc-mode. Do I have that right, too? Good.
Shall I open a bug report to joust with it, or is saying so enough?
I'd much rather the conflicting feature be disabled or made optional
in cc-mode, so that not only e-p-m starts working but also C-M-u,
C-M-f, etc. If don't agree, find a small enough solution that fixes
e-p-m, propose it here,  then find a consensus that overrides my
opinion, it happens all the time, won't hold it against you or
anyone else, good luck.

--
João Távora

[-- Attachment #2: Type: text/html, Size: 3531 bytes --]

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Kévin Le Gouguec]

João Távora <joaotavora@gmail.com> writes:

>> can you suggest some other mechanism by
>> which CC Mode can create the required font-locking?  If you can, and
>> it's workable, we all win.
>
> No, I can't, but I've asked others to chime in. I would personally do it
> (in fact I already do do it) via flymake-mode.

To expand on that a bit…

AFAICT from its documentation, flymake aims to be "a universal
on-the-fly syntax checker for GNU Emacs".  It is distributed as a core
Emacs library, and as an ELPA package requiring Emacs 26.1.

Looking at this situation from afar[1], invalid-syntax highlighting
sounds exactly like the kind of feature that could be implemented as a
flymake backend[2].

(
    Although I guess the question then moves to "How would the
    *backend* implement this feature?" to which I have no answer,
    knowing nothing of either flymake's API or CC mode's
    invalid-syntax detection process.  Abstractly, I guess that would
    imply

    - lifting some fontification code off CC mode since I assume the
      flymake frontend would handle the presentation,

    - keeping the invalid-syntax detection code,

    - marshalling the information it gathers into flymake
      "diagnostics".
)

From this excerpt of the discussion of bug#36474[3]:

Alan Mackenzie <acm@muc.de> writes:

>> 4. Someone reinvents electric-pair-mode in cc-model.el.
>> Let's not do this.
>
> No, let's not do that!  :-)

… I understand that it is a shared principle that there is no
intrinsic value to CC mode implementing features that another library
already provides[4], as that complicates maintenance.


Anyway, this was my very abstract reading of the situation; if I knew
the actual code better, I could probably come up with a truckload of
reasons why *as things stand now*, tearing the highlighting feature
apart and banging on it until it fits flymake's API is not the path of
least resistance.

Therefore, feel free to disregard.


[1] "Long-time-user-long-time-lurker-no-contribution-to-speak-of"-kind
    of afar.

[2] Quoting (flymake) Backend functions:

    >    A backend’s responsibility is to diagnose the contents of a
    > buffer for problems, registering the problem’s positions, type,
    > and summary description.  This information is collected in the
    > form of diagnostic objects created by the function
    > ‘flymake-make-diagnostic’ (*note Flymake utility functions), and
    > then handed over to Flymake, which proceeds to annotate the
    > buffer.

[3] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=36474#8

[4] Although in this particular case, CC mode strives to support Emacs ≥
    24.5, so depending on flymake would limit the feature to Emacs ≥
    26.1.

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 3907 bytes --]

On Wed, Jul 3, 2019 at 7:25 PM Kévin Le Gouguec <kevin.legouguec@gmail.com>
wrote:

> Looking at this situation from afar[1], invalid-syntax highlighting
> sounds exactly like the kind of feature that could be implemented as a
> flymake backend[2].

If you're not connecting these particular dots, I wrote Flymake (or
rather, I rewrote it) recently, so that's why I'm partial to it.  But
there is also another very good and popular solution, Flycheck, which
does basically the same thing (but is not bundled with Emacs).

Anyway, if you open an Emacs C source and type M-x flymake-mode you
should already get the full syntax validation of the buffer as performed
by the compiler.  In other projects, it's easy to setup (see the
docstring for the flymake-cc-command variable).

>     - lifting some fontification code off CC mode since I assume the
>       flymake frontend would handle the presentation,
> ...
> apart and banging on it until it fits flymake's API is not the path of
> least resistance.

It's not trivial to write a Flymake backend, but it's not hard either.
I wrote documentation for that in the manual and in docstrings. For this
, you don't need to lift any code off CC-mode (or any other mode) if it
follows basic assumptions.  I hacked up a (very lightly tested) backend
that will diagnose unterminated multi-line strings in most modes.
This includes cc-derived modes if you monkey-patch them with

   (defun c-unescaped-nls-in-string-p (&optional quote-pos) t)

...or if you find some other way to disable the problematic feature.
Here is the code.

(cl-defun flymake-check-unterminated-strings
    (report-fn &key changes-start changes-end &allow-other-keys)
  "Flymake backend for checking unterminated strings in c-like modes"
  (save-excursion
    (cl-loop with collected = (list)
             with start = (or changes-start (point-min))
             with start-ppss = (syntax-ppss)
             with end = (or changes-end (point-max))
             initially (goto-char start)
             (when (nth 3 start-ppss)
               (goto-char (setq start (nth 8 start-ppss)))
               (setq start-ppss (syntax-ppss)))
             for ppss = start-ppss then (syntax-ppss)
             for next = (if (and (nth 3 ppss)
                                 (nth 8 ppss))
                            (let* ((lep (line-end-position))
                                   (probe (char-before lep))
                                   (string-end (ignore-errors
                                                 (1+ (scan-sexps (nth 8
ppss) 1)))))
                              (setq end (or string-end (point-max)))
                              (cond
                                ((eq probe ?\\) (1+ lep))
                                ((and (not (= lep (point))) (eq probe ?\"))
                                 (and string-end
                                      (1+ string-end)))
                                (t (push (flymake-make-diagnostic
                                          (current-buffer) (point)
                                          (1+ lep) :error "unterminated
string")
                                         collected)
                                   (and string-end (1+ string-end)))))
                          (1+ (point)))
             while (and next
                        (<= next end))
             do (goto-char next)
             finally (funcall report-fn
                              collected
                              :region (cons start end))
             (cl-return collected))))

You can try it with

  (add-hook 'flymake-diagnostic-functions
#'flymake-check-unterminated-strings nil t)

You might have to turn flymake off and on again.  You also need a very
recent Flymake (version 1.0.8 should be in GNU ELPA shortly).

--
João Távora

[-- Attachment #2: Type: text/html, Size: 4759 bytes --]

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Kévin Le Gouguec]

João Távora <joaotavora@gmail.com> writes:

> If you're not connecting these particular dots, I wrote Flymake (or
> rather, I rewrote it) recently, so that's why I'm partial to it.  But
> there is also another very good and popular solution, Flycheck, which
> does basically the same thing (but is not bundled with Emacs).

(Yup, the dots were connected.  I know about Flycheck and I did notice
you picking up Flymake recently, for which I am quite grateful!)

The main reason *I* am partial to Flymake, as a user (and in the context
of this feature of CC mode), is because it is bundled with Emacs (has
been for 15 years).  This means I can think of incorrect-code
highlighting as a "core feature", as Alan called it (though it's core as
in "part of the Emacs core", not necessarily as in "implemented by the
major mode and impossible to toggle off").

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Alan Mackenzie]

Hello, João.

On Wed, Jul 03, 2019 at 14:31:58 +0100, João Távora wrote:
> On Wed, Jul 3, 2019 at 11:58 AM Alan Mackenzie <acm@muc.de> wrote:

[ Bug #36474 ]

> You believe this is a bug in e-p-m. It's certainly your right to believe
> that. I believe it's a bug in cc-mode. Do I have that right, too?

No, you don't.  What you're implying is that as maintainer of a minor
mode, you have the right to dictate which parts of Emacs may be used by
a major mode, and how.  That's not how things work.

And I have not said there is a bug "in" electric-pair-mode.  I said that
it does not offer the requisite facilities to support CC Mode.  I have
been asking you for an enhancement rather than a bug fix.

> Good.

[ .... ]

> If don't agree, find a small enough solution that fixes e-p-m, propose
> it here,  then find a consensus that overrides my opinion, it happens
> all the time, won't hold it against you or anyone else, good luck.

I'm prepared to do this, but only on the understanding you will, on
principle, accept my solution, subject only to the customary review and
amendments process that any change goes through.  

What do you say?

> --
> João Távora

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 139 bytes --]

On Thu, Jul 4, 2019 at 4:05 PM Alan Mackenzie <acm@muc.de> wrote:

> What do you say?

I say we should explore alternatives.

João

[-- Attachment #2: Type: text/html, Size: 268 bytes --]

[PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Alan Mackenzie]

Hello, Joãa.

On Wed, Jul 03, 2019 at 14:31:58 +0100, João Távora wrote:
[ .... ]

> If [you] don't agree, find a small enough solution that fixes e-p-m,
> propose it here,  then find a consensus that overrides my opinion, it
> happens all the time, won't hold it against you or anyone else, good
> luck.

Here is the entire part of the fix needed for bug #36474, that should go
into elec-pair-mode.el:


diff --git a/lisp/elec-pair.el b/lisp/elec-pair.el
index 5fb9d751e2..ba86b55167 100644
--- a/lisp/elec-pair.el
+++ b/lisp/elec-pair.el
@@ -484,6 +484,23 @@ electric-pair-default-inhibit
       (electric-pair-inhibit-if-helps-balance char)
     (electric-pair-conservative-inhibit char)))
 
+(defun electric-pair-cc-inhibit-predicate (char)
+  "Return t to inhibit the insertion of a second copy of CHAR.
+
+At the time of call, point is just after the newly inserted CHAR.
+
+When CHAR is \", t will be returned unless the \" is marked with
+a string fence syntax-table text property.  For other characters
+`electric-pair-default-inhibit' is called and its value returned.
+
+This function is the appropriate value of
+`electric-pair-inhibit-predicate' for CC Mode modes, which mark
+invalid strings with such a syntax table text property on the
+opening \" and the next unescaped end of line."
+  (if (eq char ?\")
+      (not (equal (get-text-property (1- (point)) 'syntax-table) '(15)))
+    (electric-pair-default-inhibit char)))
+
 (defun electric-pair-post-self-insert-function ()
   (let* ((pos (and electric-pair-mode (electric--after-char-pos)))
          (skip-whitespace-info))


You will note that it is the mere addition of a function, not used
anywhere else in elec-pair.el, so cannot possibly damage its
functionality.  If I was being perverse, I could even put this function
in cc-mode.el instead, though this would not be good practice.

This new function has been tested, and it works.

Now, will you please just decide to install this patch.  If you like, I
can write a commit message for it.

> --
> João Távora

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 718 bytes --]

On Wed, Jul 03, 2019 at 14:31:58 +0100, João Távora wrote:

> [ .... ]
>
> > If [you] don't agree, find a small enough solution that fixes e-p-m,
> > propose it here,  then find a consensus that overrides my opinion, it
> > happens all the time, won't hold it against you or anyone else, good
> > luck.
>
> Here is the entire part of the fix needed for bug #36474, that should go
> into elec-pair-mode.el:


I'll have a look, but please show the whole change, not just
the new function. A new function by itself has no effect in e-p-m or
any other part of emacs. You should show where you intend to use it
(it should, in principle, be in one of e-p-m's external-facing variables).

Thanks,
João

[-- Attachment #2: Type: text/html, Size: 1066 bytes --]

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Alan Mackenzie]

Hello, João.

On Thu, Jul 04, 2019 at 19:45:26 +0100, João Távora wrote:
> On Wed, Jul 03, 2019 at 14:31:58 +0100, João Távora wrote:

> > [ .... ]

> > > If [you] don't agree, find a small enough solution that fixes e-p-m,
> > > propose it here,  then find a consensus that overrides my opinion, it
> > > happens all the time, won't hold it against you or anyone else, good
> > > luck.

> > Here is the entire part of the fix needed for bug #36474, that should go
> > into elec-pair-mode.el:


> I'll have a look, but please show the whole change, not just
> the new function. A new function by itself has no effect in e-p-m or
> any other part of emacs. You should show where you intend to use it
> (it should, in principle, be in one of e-p-m's external-facing variables).

At the top of c-basic-common-init:

diff -r eb4ee9bb0682 cc-mode.el
--- a/cc-mode.el	Thu Jul 04 13:01:08 2019 +0000
+++ b/cc-mode.el	Thu Jul 04 18:56:49 2019 +0000
@@ -521,6 +521,14 @@
 
   (setq c-buffer-is-cc-mode mode)
 
+  ;; Prepare for the use of `electric-pair-mode':
+  (eval-after-load "elec-pair"
+    '(when (and (boundp 'electric-pair-inhibit-predicate)
+		(fboundp 'electric-pair-cc-inhibit-predicate))
+       (make-local-variable 'electric-pair-inhibit-predicate)
+       (setq electric-pair-inhibit-predicate
+	     #'electric-pair-cc-inhibit-predicate)))
+
   ;; these variables should always be buffer local; they do not affect
   ;; indentation style.
   (make-local-variable 'parse-sexp-ignore-comments)

> Thanks,
> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1135 bytes --]

On Thu, Jul 4, 2019 at 8:01 PM Alan Mackenzie <acm@muc.de> wrote:

> +  ;; Prepare for the use of `electric-pair-mode':
> +  (eval-after-load "elec-pair"
> +    '(when (and (boundp 'electric-pair-inhibit-predicate)
> +               (fboundp 'electric-pair-cc-inhibit-predicate))
> +       (make-local-variable 'electric-pair-inhibit-predicate)
> +       (setq electric-pair-inhibit-predicate
> +            #'electric-pair-cc-inhibit-predicate)))
>

What you are doing here is perfectly legal.  I designed the
electric-pair-inhibit-predicate variable to be external to users and
other modes, presuming those users know what they are doing.
You can use add-function or setq the variable directly. I prefer
the former, but it's probably not available in emacs 24.  It's
your choice.

The function you put there shouldn't be in the elec-pair.el
library, so I suggest you put it in cc-mode.el or wherever you
feel it fits.  It'll probably save you that fboundp check.

Naturally, this doesn't fix the C-M-* navigation in multi-line
strings (as you know, if that were fixed, you woundn't need
this tweak)

João

[-- Attachment #2: Type: text/html, Size: 1536 bytes --]

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Alan Mackenzie]

Hello, João.

On Thu, Jul 04, 2019 at 22:44:36 +0100, João Távora wrote:
> On Thu, Jul 4, 2019 at 8:01 PM Alan Mackenzie <acm@muc.de> wrote:

> > +  ;; Prepare for the use of `electric-pair-mode':
> > +  (eval-after-load "elec-pair"
> > +    '(when (and (boundp 'electric-pair-inhibit-predicate)
> > +               (fboundp 'electric-pair-cc-inhibit-predicate))
> > +       (make-local-variable 'electric-pair-inhibit-predicate)
> > +       (setq electric-pair-inhibit-predicate
> > +            #'electric-pair-cc-inhibit-predicate)))
> >

> What you are doing here is perfectly legal.  I designed the
> electric-pair-inhibit-predicate variable to be external to users and
> other modes, presuming those users know what they are doing.
> You can use add-function or setq the variable directly. I prefer
> the former, but it's probably not available in emacs 24.  It's
> your choice.

I've done this, and closed bug #36474.

> The function you put there shouldn't be in the elec-pair.el
> library, so I suggest you put it in cc-mode.el or wherever you
> feel it fits.  It'll probably save you that fboundp check.

Yes.

> Naturally, this doesn't fix the C-M-* navigation in multi-line
> strings (as you know, if that were fixed, you woundn't need
> this tweak)

I have an idea how to make C-M-* work in these bogus multi-line strings,
but it's going to take more than a couple of days to implement.  It
involves only having the pertinent string fence syntax-table text
properties present during fontification and CC Mode's
before/after-change-functions.

> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 919 bytes --]

On Mon, Jul 8, 2019 at 11:05 AM Alan Mackenzie <acm@muc.de> wrote:

> I have an idea how to make C-M-* work in these bogus multi-line strings,
> but it's going to take more than a couple of days to implement.  It
> involves only having the pertinent string fence syntax-table text
> properties present during fontification and CC Mode's
> before/after-change-functions.

No idea if that approach has any pitfalls, but if it doesn't, it's a good
bet that you won't need the electric-pair-inhibit-predicate tweak
you just committed. You'll probably also unbreak a
C++/electric-pair test, bringing back cc-mode closer to what is was
before the pursuit of these new fontification features.

Nevertheless I think you should heed Stefan's advice. It should give
you all of the above (invalid string indicator included), with much
less code and hassle, and you can use string fences some other time.

João

[-- Attachment #2: Type: text/html, Size: 1060 bytes --]

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Stefan Monnier]

> I have an idea how to make C-M-* work in these bogus multi-line strings,
> but it's going to take more than a couple of days to implement.  It
> involves only having the pertinent string fence syntax-table text
> properties present during fontification and CC Mode's
> before/after-change-functions.

While I'm not completely sure what this will look like, my gut reaction
to this description is "Yuck!!  Ugly hack to workaround breakage
introduced by other ugly hack!"
Usually this leads to yet more ugly hacks.


        Stefan

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Stefan Monnier]

> While I'm not completely sure what this will look like, my gut reaction
> to this description is "Yuck!!  Ugly hack to workaround breakage
> introduced by other ugly hack!"
> Usually this leads to yet more ugly hacks.

BTW, re-reading this, I see it could be taken as a personal attack, so
I want to clarify that I consider Emacs to be full of ugly hacks and
that a significant proportion of them come directly from me:
sometimes it's still the best we can come up with :-(

But I obviously still think in this case that the choice of leaving the
text on subsequent lines (after a string with no \ nor closing ")
highlighted as if they were part of the string is a much better option.


        Stefan

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Alan Mackenzie]

Hello, Stefan.

On Mon, Jul 08, 2019 at 12:33:56 -0400, Stefan Monnier wrote:
> > While I'm not completely sure what this will look like, my gut reaction
> > to this description is "Yuck!!  Ugly hack to workaround breakage
> > introduced by other ugly hack!"
> > Usually this leads to yet more ugly hacks.

> BTW, re-reading this, I see it could be taken as a personal attack, so
> I want to clarify that I consider Emacs to be full of ugly hacks and
> that a significant proportion of them come directly from me:
> sometimes it's still the best we can come up with :-(

> But I obviously still think in this case that the choice of leaving the
> text on subsequent lines (after a string with no \ nor closing ")
> highlighted as if they were part of the string is a much better option.

No it is not.  I explained this in detail last Thursday, to which your
reply was to snip my explanation, apparently without reading it.  You
then posed another point to which what you had snipped containe the
answer.

If you really believe that the old way is better, then I suggest you
take up that post of mine from last Thursday and actually debate the
issues with me.

The fact that it is difficult to get CC Mode's fontification in most
other modes which are constrained by the restrictive
syntax-propertize-function mechanism is not a good argument against CC
Mode; rather it should act as a spur to remove those restrictions from
those other modes.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Stefan Monnier]

> If you really believe that the old way is better, then I suggest you
> take up that post of mine from last Thursday and actually debate the
> issues with me.

I don't see anything to debate here.  We just have different
preferences, neither of which is technically right or wrong.


        Stefan

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Alan Mackenzie]

Hello, Stefan.

On Mon, Jul 08, 2019 at 11:49:54 -0400, Stefan Monnier wrote:
> > I have an idea how to make C-M-* work in these bogus multi-line strings,
> > but it's going to take more than a couple of days to implement.  It
> > involves only having the pertinent string fence syntax-table text
> > properties present during fontification and CC Mode's
> > before/after-change-functions.

> While I'm not completely sure what this will look like, my gut reaction
> to this description is "Yuck!!  Ugly hack to workaround breakage
> introduced by other ugly hack!"

Don't knock it - it's working, and will give João what he wants whilst
preserving correct fontification.  Although it is not yet in a state
where I want to post a patch on this list.

[ .... ]

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Stefan Monnier]

> Don't knock it - it's working, and will give João what he wants whilst
> preserving correct fontification.

Several people have already pointed out that what you call here "correct
fontification" is "incorrect" in the cases where the error is a missing
\ rather than a missing ".

So it would be less adversarial for you to write "the fontification I'm
looking for" or something like that.


        Stefan

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Alan Mackenzie]

Hello, Stefan.

On Mon, Jul 08, 2019 at 13:29:46 -0400, Stefan Monnier wrote:
> > Don't knock it - it's working, and will give João what he wants whilst
> > preserving correct fontification.

> Several people have already pointed out that what you call here "correct
> fontification" is "incorrect" in the cases where the error is a missing
> \ rather than a missing ".

"Several people" is just one that I have seen, and he is wrong.  Whether
the error "is" a missing \ or a missing " (for whatever value of "is"),
the purpose of the fontification is to draw attention to this error.
This error is at the EOL where there is a missing " or \.  In that sense
CC Mode's fontification of it is correct.

> So it would be less adversarial for you to write "the fontification I'm
> looking for" or something like that.

I'm faced with an "opponent" who snipped my careful argument without
appearing to have read it, and who is unprepared to counter it on its
merits, yet insists it is wrong.

How can you expect me to be other than adversarial when faced with such
an attitude?

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Stefan Monnier]

>> > Don't knock it - it's working, and will give João what he wants whilst
>> > preserving correct fontification.
>> Several people have already pointed out that what you call here "correct
>> fontification" is "incorrect" in the cases where the error is a missing
>> \ rather than a missing ".
> "Several people" is just one that I have seen, and he is wrong.

João gave you actual examples where fontifying the second line as
a code rather than string was an error.  So we're at least 2.

> Whether
> the error "is" a missing \ or a missing " (for whatever value of "is"),
> the purpose of the fontification is to draw attention to this error.
> This error is at the EOL where there is a missing " or \.  In that sense
> CC Mode's fontification of it is correct.

Maybe we're not talking about the same thing: The addition of some kind
of error/warning face on this line is indeed always correct and is
something that can easily be done without any need to fiddle with
syntax-table text properties.  I'm definitely not arguing against this
highlighting of the immediate error.

But I'm talking about is the highlighting of the subsequent line(s).

You seem to want them to be highlighted as if the next line was not part
of a string (i.e. assume that a " was missing on the previous line),
whereas all other major modes highlight that second line under the
assumption that what was missing was a final \ on the previous line (or
whatever other syntax the corresponding language requires).

Both are wrong sometimes and right at other times.

> How can you expect me to be other than adversarial when faced with such
> an attitude?

I guess I like to expect the unexpected ;-)


        Stefan

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Clément Pit-Claudel]

On 2019-07-08 16:59, Stefan Monnier wrote:
> You seem to want them to be highlighted as if the next line was not part
> of a string (i.e. assume that a " was missing on the previous line),
> whereas all other major modes highlight that second line under the
> assumption that what was missing was a final \ on the previous line (or
> whatever other syntax the corresponding language requires).
> 
> Both are wrong sometimes and right at other times.

I imagine this is less of a problem for people who use modes that automatically insert string delimiters (i.e. when you press ", a second matching quote immediately appears, so you don't run into fontification issues).

I don't use these, but over time Emacs has taught me to always write two double quotes at once, or else my entire buffer will at once turn a uniform brown color.  I find the effect pretty distracting, and I'd much prefer for Emacs (in all programming modes, in fact) to assume that my unterminated strings end at the end of the line.  That's not more "correct", of course, but it does minimize the amount of refontification and blinking in a pretty common case.

(Btw, I wish there was a similarly easy way to minimize the Flycheck version of the problem, where often a small syntax mistake will turn the entire buffer into sea of red squiggles)

Clément.

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1100 bytes --]

On Tue, Jul 9, 2019 at 7:42 AM Clément Pit-Claudel <cpitclaudel@gmail.com>
wrote:
>
> of the line.  That's not more "correct", of course, but it does minimize
the amount
> of refontification and blinking in a pretty common case.

No, it doesn't. Suppose you had that hypothetical version of Emacs and
you did have a multi-line string, properly escaped with backslashes. Now
you
removed the backslash. Brutal "wrong" refontification ensues just as now,
fontifying as code what up to now used to be fontified as a string.

Just as distracting, really.

Both situations are wrong, and neither is "more wrong" than the other.
But the version you propose is much harder to implement, likely less
efficient, and backward incompatible to how Emacs has worked over
many, many years. Depending on how it is implemented (certainly how Alan
implemented it) it breaks things in Emacs core and third-party code. It is
also
useless in languages which do allow unescaped multi-line strings (which
I think are the norm nowadays, but I'm not sure).

Overall, clearly not a net win.

João

[-- Attachment #2: Type: text/html, Size: 1550 bytes --]

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 395 bytes --]

On Tue, Jul 9, 2019 at 10:06 AM João Távora <joaotavora@gmail.com> wrote:

> many, many years. Depending on how it is implemented (certainly how Alan
> implemented it) it breaks things in Emacs core and third-party code.

Just a small followup: one of the things it breaks is precisely a tool
which
helps you to more predictably fix the situation, electric-pair-mode.

João Távora

[-- Attachment #2: Type: text/html, Size: 522 bytes --]

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Alan Mackenzie]

Hello, João.

On Tue, Jul 09, 2019 at 10:23:39 +0100, João Távora wrote:
> On Tue, Jul 9, 2019 at 10:06 AM João Távora <joaotavora@gmail.com> wrote:

> > many, many years. Depending on how it is implemented (certainly how Alan
> > implemented it) it breaks things in Emacs core and third-party code.

Can we perhaps keep the disparagement a bit more muted, please?

I'm busy implementing what you've said you want, namely the ability to
use the C-M-* commands across "strings" consisting of two "s with at
least one unescaped NL between them.

At the moment, in the following:

    printf ("Hello
             world");

, with point on the first ", C-M-f goes to after the second ".  With
point on the w, C-M-u goest to the first ".

There are one or two tricky edge cases I'm still working on.

> Just a small followup: one of the things it breaks is precisely a tool
> which helps you to more predictably fix the situation,
> electric-pair-mode.

As I asked, can we keep the disparagement down, please?  It's not
helping anything or anybody, particularly not me.

Thanks.

> João Távora

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 946 bytes --]

On Tue, Jul 9, 2019 at 10:52 AM Alan Mackenzie <acm@muc.de> wrote:

> Hello, João.
>
> > > many, many years. Depending on how it is implemented (certainly how
> Alan
> > > implemented it) it breaks things in Emacs core and third-party code.
> Can we perhaps keep the disparagement a bit more muted, please?
>

How can this be disparagement? It's a simple statement of fact. If you
didn't
break things, what did you fix and what are you fixing?  Is "break" the
word that you associate with disparagement, maybe? I admit it's a word
that I hear very often in software engineering, I didn't invent it.

Maybe you associate it with clumsiness, like I'm implying you're a
doofus breaking things recklessly. I'm not, believe me. But unless you
give me a better word that expresses "take action in area A with the
consequence that the previous behavior in area B isn't observed any
more", I'll have to keep using "break".

João

[-- Attachment #2: Type: text/html, Size: 1466 bytes --]

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 598 bytes --]

On Tue, Jul 9, 2019 at 11:54 AM João Távora <joaotavora@gmail.com> wrote:


> Maybe you associate it with clumsiness, like I'm implying you're a
> doofus breaking things recklessly. I'm not, believe me. But unless you
> give me a better word that expresses "take action in area A with the
> consequence that the previous behavior in area B isn't observed any
> more", I'll have to keep using "break".
>

Sorry, I meant "..the previous desirable behavior in area B..". That equates
to "break".  If it's "...the previous undesirable behaviour", the word is
"fix",
obviously.

João

[-- Attachment #2: Type: text/html, Size: 1120 bytes --]

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Dmitry Gutov]

On 09.07.2019 13:54, João Távora wrote:

> How can this be disparagement? It's a simple statement of fact. If you 
> didn't
> break things, what did you fix and what are you fixing?  Is "break" the
> word that you associate with disparagement, maybe? I admit it's a word
> that I hear very often in software engineering, I didn't invent it.

Since the "correct" behavior is a matter of interpretation here, who 
broke what is a matter of interpretation. And I'm sure you can agree 
that "you broke XXX" can be interpreted emotionally. There's no real 
need to repeat that a few times, especially while certain someone is 
working on "fixing" it.

To get back to Clement's question (and maybe I'm just restating your 
opinion here), I think:

* Fontifying "broken" strings only until EOL might be beneficial, at 
least in certain languages, where it's consistent with the syntax. The 
result will be less "blinking".

* It doesn't seem trivial to implement without breaking a lot of 
pair-matching and quote-matching functionality, because both font-lock 
and the latter features depend on parse status, and buffer can be 
fontified in chunks, etc.

contextual refontification (was: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode)

[Stefan Monnier]

> * Fontifying "broken" strings only until EOL might be beneficial, at least
> in certain languages, where it's consistent with the syntax. The result will
> be less "blinking".

This problem is not specific to strings: it also affects comments.

BTW, the current default behavior delays re-fontifying the rest of the
buffer (so called "contextual refontification") by jit-lock-context-time
(by default 0.5s) which can sometimes avoid the flashing.

It was not implemented specifically for that purpose, but it does
sometimes have that effect.  Maybe we could develop a hack for font-lock
which would do it in a less "accidental" way.

IOW make it so that font-lock:
- delays refontification after the current line if the end of line is
  now within a string/comment (and wasn't before).
- "un-delays" this fontification if the end of line is now outside of any
  string/comment (but was within it before).
- of course, with some kind of timeout, but one longer than 0.5s.

Or something like that.


        Stefan

Re: contextual refontification (was: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode)

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 4412 bytes --]

On Tue, Jul 9, 2019 at 4:40 PM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> IOW make it so that font-lock:
> - delays refontification after the current line if the end of line is
>   now within a string/comment (and wasn't before).
> - "un-delays" this fontification if the end of line is now outside of any
>   string/comment (but was within it before).
> - of course, with some kind of timeout, but one longer than 0.5s.
>
> Or something like that.


Here's my take on that. Clément/Dmitry if you find the time test this. ..
It should, in principle, reduce "blinking" in all modes (for people who
aren't using electric-pair-mode or something like that). If it doesn't,
I've made some kind of false assumption.

Btw to restart jit-lock-mode in a running emacs, you need to
do sth like this:

(progn
  (jit-lock-mode -1)
  (cancel-timer jit-lock-context-timer)
  (setq jit-lock-context-timer nil)
  (jit-lock-mode 1))

João

diff --git a/lisp/jit-lock.el b/lisp/jit-lock.el
index 48998a81fe..0d7c434f3f 100644
--- a/lisp/jit-lock.el
+++ b/lisp/jit-lock.el
@@ -123,6 +123,15 @@ jit-lock-context-time
   :type '(number :tag "seconds")
   :group 'jit-lock)

+(defcustom jit-lock-multiline-string-grace 2
+  "Like `jit-lock-context-time' but for unterminated multiline strings.
+If the user has just opened an unterminated string at EOL, give
+him/her some grace time before deciding it is a multi-line string
+and fontifying accordingly, do so only if the user stares idle at
+that string for more than this many seconds."
+  :type '(number :tag "seconds")
+  :group 'jit-lock)
+
 (defcustom jit-lock-defer-time nil ;; 0.25
   "Idle time after which deferred fontification should take place.
 If nil, fontification is not deferred.
@@ -232,7 +241,7 @@ jit-lock-mode
       (unless jit-lock-context-timer
         (setq jit-lock-context-timer
               (run-with-idle-timer jit-lock-context-time t
-                                   'jit-lock-context-fontify)))
+                                   (jit--lock-context-timer-function))))
       (setq jit-lock-context-unfontify-pos
             (or jit-lock-context-unfontify-pos (point-max))))

@@ -306,6 +315,44 @@ jit-lock--debug-fontify
                                    pos 'fontified)))))))))
       (setq jit-lock-defer-buffers nil))))

+(defun jit--lock-context-timer-function ()
+  (let (last            ; point marker the last time context timer was run
+        in-s-or-c-p     ; t if in string or comment that time around
+        grace-timer  ; idle timer for fontifying unterminated s-or-c, or
nil
+        )
+    (lambda ()
+      (let ((point (point-marker))
+            (new-in-s-or-c-p
+             (nth 8 (save-excursion (syntax-ppss (line-end-position))))))
+        (if (and jit-lock-multiline-string-grace
+                 last
+                 (eq (marker-buffer last) (current-buffer))
+                 (eq (line-number-at-pos last) (line-number-at-pos)))
+            (cond ((and (null in-s-or-c-p) new-in-s-or-c-p (null
grace-timer))
+                   (setq grace-timer
+                         (run-with-idle-timer
jit-lock-multiline-string-grace nil
+                                              (lambda ()
+                                                (jit-lock-context-fontify)
+                                                (setq grace-timer nil)))))
+                  ((and in-s-or-c-p
+                        (null new-in-s-or-c-p)
+                        grace-timer)
+                   (cancel-timer grace-timer)
+                   (setq grace-timer nil))
+                  (t
+                   ;; no state change, leave everything as it was
+                   ))
+          ;; left the line somehow or customized feature away: cancel
+          ;; everything, resume normal operation.
+          (when grace-timer
+            (cancel-timer grace-timer)
+            (setq grace-timer nil)))
+        ;; proceed as usual, unless grace-timer is counting
+        (unless grace-timer
+          (jit-lock-context-fontify))
+        (setq last point in-s-or-c-p new-in-s-or-c-p)))))
+
+
 (defun jit-lock-register (fun &optional contextual)
   "Register FUN as a fontification function to be called in this buffer.
 FUN will be called with two arguments START and END indicating the region

[-- Attachment #2: Type: text/html, Size: 5906 bytes --]

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2188 bytes --]

On Tue, Jul 9, 2019 at 4:19 PM Dmitry Gutov <dgutov@yandex.ru> wrote:

> Since the "correct" behavior is a matter of interpretation here, who
> broke what is a matter of interpretation. And I'm sure you can agree
> that "you broke XXX" can be interpreted emotionally.

Anything can (and everything is). That's the nature of being an animal
and not a machine.

So what word/phrasing would you suggest when you need to
unequivocally state that "the actions (of individual A) led to the loss of
a previously verified behaviour X, desired by individual B, such that both
A and B now agree that X is desirable and A is working to
make X work again"?

I don't repeat this gratuitously: I am using it to make an argument
against a feature, listing it in the "cons" part. Other people are
stating facts that they add to the "pros".

> To get back to Clement's question (and maybe I'm just restating your
> opinion here), I think:
>
> * Fontifying "broken" strings only until EOL might be beneficial, at
> least in certain languages, where it's consistent with the syntax. The
> result will be less "blinking".

Depends on how you edit. If the most common case is that
you, for example, C-k kill-line a closing " at EOL away, then yes, you're
right. If the most common case is that you RET newline at the middle
of a string then no, you're wrong. Both can be accidents/mistakes
or deliberate actions. But let's say they are accidents. Which
one is more common? For me it's the RET thing, after which I
usually C-M-u C-M-SPC to fix the situation. So I would experience
_more_ blinking (along with all the other disadvantages).

Consistency with how the compiler views invalid code is not very
important for me, because every view allows me to spot the error
unequivocally (also the compiler probably doesn't care about
this because the compiler doesn't edit code or fontify it).

> * It doesn't seem trivial to implement without breaking a lot of
> pair-matching and quote-matching functionality, because both font-lock
> and the latter features depend on parse status, and buffer can be
> fontified in chunks, etc.

I would agree fully here.

João

[-- Attachment #2: Type: text/html, Size: 2901 bytes --]

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Alan Mackenzie]

Hello, João.

On Tue, Jul 09, 2019 at 11:54:39 +0100, João Távora wrote:
> On Tue, Jul 9, 2019 at 10:52 AM Alan Mackenzie <acm@muc.de> wrote:

> > Hello, João.

> > > > many, many years. Depending on how it is implemented (certainly how
> > > > Alan implemented it) it breaks things in Emacs core and
> > > > third-party code.
> > Can we perhaps keep the disparagement a bit more muted, please?


> How can this be disparagement? It's a simple statement of fact.

It is an extreme interpretation of somewhat controversial facts.

> If you didn't break things, what did you fix and what are you fixing?

I'm fixing electric-pair-mode, and I'm adding a feature to CC Mode
specially for you, because you've asked for it so insistently.  :-)

[ .... ]

So, although the enhancement is not yet bug free, I'm supplying you with
the following patch, with the request you undo your unofficial
modifications to CC Mode and try out the patch.  It is not 100% bug free,
but might be close to it.

It should allow you to do syntactic operations on two disjoint "s, while
at the same time preserving CC Mode's fontification strategy.

You may have less justification for complaining after trying this out.
No apologies for that.  ;-)



diff --git a/lisp/progmodes/cc-defs.el b/lisp/progmodes/cc-defs.el
index a43f1ac72d..ad3cb9c125 100644
--- a/lisp/progmodes/cc-defs.el
+++ b/lisp/progmodes/cc-defs.el
@@ -406,6 +406,25 @@ lookup-syntax-properties
         (forward-sexp)
         (= (point) (+ 4 (point-min)))))))
 
+(defmacro c-is-escaped (pos)
+  ;; Are there an odd number of backslashes before POS?
+  `(save-excursion
+     (goto-char ,pos)
+     (not (zerop (logand (skip-chars-backward "\\\\") 1)))))
+
+(defmacro c-will-be-escaped (pos beg end)
+  ;; Will the character after POS be escaped after the removal of (BEG END)?
+  ;; It is assumed that (>= POS END).
+  `(save-excursion
+     (let ((-end- ,end)
+	   count)
+       (goto-char ,pos)
+       (setq count (skip-chars-backward "\\\\" -end-))
+       (when (eq (point) -end-)
+	 (goto-char ,beg)
+	 (setq count (+ count (skip-chars-backward "\\\\"))))
+       (not (zerop (logand count 1))))))
+
 (defvar c-use-extents)
 
 (defmacro c-next-single-property-change (position prop &optional object limit)
@@ -1019,6 +1038,14 @@ c-major-mode-is
 ;; properties set on a single character and that never spread to any
 ;; other characters.
 
+(defmacro c-put-syn-tab (pos value)
+  ;; Set both the syntax-table and the c-fl-syn-tab text properties at POS to
+  ;; VALUE (which should not be nil).
+  `(let ((-pos- ,pos)
+	 (-value- ,value))
+     (c-put-char-property -pos- 'syntax-table -value-)
+     (c-put-char-property -pos- 'c-fl-syn-tab -value-)))
+
 (eval-and-compile
   ;; Constant used at compile time to decide whether or not to use
   ;; XEmacs extents.  Check all the extent functions we'll use since
@@ -1146,6 +1173,12 @@ c-clear-char-property
 	 ;; Emacs < 21.
 	 `(c-clear-char-property-fun ,pos ',property))))
 
+(defmacro c-clear-syn-tab (pos)
+  ;; Remove both the 'syntax-table and `c-fl-syn-tab properties at POS.
+  `(let ((-pos- ,pos))
+     (c-clear-char-property -pos- 'syntax-table)
+     (c-clear-char-property -pos- 'c-fl-syn-tab)))
+
 (defmacro c-min-property-position (from to property)
   ;; Return the first position in the range [FROM to) where the text property
   ;; PROPERTY is set, or `most-positive-fixnum' if there is no such position.
@@ -1381,6 +1414,29 @@ c-put-char-properties-on-char
 	     `((setq c-syntax-table-hwm (min c-syntax-table-hwm (point)))))
 	 (c-put-char-property (point) ,property ,value)
 	 (forward-char)))))
+
+(defmacro c-with-extended-string-fences (beg end &rest body)
+  ;; If needed, extend the region with "mirrored" c-fl-syn-tab properties to
+  ;; contain the region (BEG END), then evaluate BODY.  If this mirrored
+  ;; region was initially empty, restore it afterwards.
+  `(let ((-beg- ,beg)
+	 (-end- ,end)
+	 )
+     (cond
+      ((null c-fl-syn-tab-region)
+       (unwind-protect
+	   (progn
+	     (c-restore-string-fences -beg- -end-)
+	     ,@body)
+	 (c-clear-string-fences)))
+      ((and (>= -beg- (car c-fl-syn-tab-region))
+	    (<= -end- (cdr c-fl-syn-tab-region)))
+       ,@body)
+      (t				; Crudely extend the mirrored region.
+       (setq -beg- (min -beg- (car c-fl-syn-tab-region))
+	     -end- (max -end- (cdr c-fl-syn-tab-region)))
+       (c-restore-string-fences -beg- -end-)
+       ,@body))))
 \f
 ;; Macros to put overlays (Emacs) or extents (XEmacs) on buffer text.
 ;; For our purposes, these are characterized by being possible to
@@ -1463,6 +1519,7 @@ c-clear-char-property-with-value
 (def-edebug-spec c-clear-char-property-with-value-on-char t)
 (def-edebug-spec c-put-char-properties-on-char t)
 (def-edebug-spec c-clear-char-properties t)
+(def-edebug-spec c-with-extended-string-fences (form form body))
 (def-edebug-spec c-put-overlay t)
 (def-edebug-spec c-delete-overlay t)
 (def-edebug-spec c-mark-<-as-paren t)
diff --git a/lisp/progmodes/cc-engine.el b/lisp/progmodes/cc-engine.el
index 40a3b72f6a..656dfd126c 100644
--- a/lisp/progmodes/cc-engine.el
+++ b/lisp/progmodes/cc-engine.el
@@ -155,6 +155,9 @@
 (defvar c-doc-line-join-re)
 (defvar c-doc-bright-comment-start-re)
 (defvar c-doc-line-join-end-ch)
+(defvar c-fl-syn-tab-region)
+(cc-bytecomp-defun c-clear-string-fences)
+(cc-bytecomp-defun c-restore-string-fences)
 
 \f
 ;; Make declarations for all the `c-lang-defvar' variables in cc-langs.
@@ -2816,7 +2819,14 @@ c-semi-pp-to-literal
 				 c-block-comment-awkward-chars)))
 		 (and (nth 4 s) (nth 7 s) ; Line comment
 		      (not (memq (char-before here) '(?\\ ?\n)))))))
-	    (setq s (parse-partial-sexp pos here nil nil s)))
+;;;; OLD STOUGH, 2019-07-09
+	    ;; (setq s (parse-partial-sexp pos here nil nil s))
+;;;; NEW STOUGH, 2019-07-09
+	    (c-with-extended-string-fences
+	     pos here
+	     (setq s (parse-partial-sexp pos here nil nil s)))
+;;;; END OF NEW STOUGH
+							    )
 	  (when (not (eq near-pos here))
 	    (c-semi-put-near-cache-entry here s))
 	  (cond
diff --git a/lisp/progmodes/cc-mode.el b/lisp/progmodes/cc-mode.el
index 98b8385fcc..395a6b1a9d 100644
--- a/lisp/progmodes/cc-mode.el
+++ b/lisp/progmodes/cc-mode.el
@@ -181,6 +181,7 @@ c-leave-cc-mode-mode
       (c-save-buffer-state ()
 	(c-clear-char-properties (point-min) (point-max) 'category)
 	(c-clear-char-properties (point-min) (point-max) 'syntax-table)
+	(c-clear-char-properties (point-min) (point-max) 'c-fl-syn-tab)
 	(c-clear-char-properties (point-min) (point-max) 'c-is-sws)
 	(c-clear-char-properties (point-min) (point-max) 'c-in-sws)
 	(c-clear-char-properties (point-min) (point-max) 'c-type)
@@ -1016,6 +1017,7 @@ c-depropertize-new-text
   (c-save-buffer-state ()
     (when (> end beg)
       (c-clear-char-properties beg end 'syntax-table)
+      (c-clear-char-properties beg end 'c-fl-syn-tab)
       (c-clear-char-properties beg end 'category)
       (c-clear-char-properties beg end 'c-is-sws)
       (c-clear-char-properties beg end 'c-in-sws)
@@ -1205,6 +1207,43 @@ c-multiline-string-check-final-quote
 	(c-put-char-property (1- (point)) 'syntax-table '(15)))
        (t nil)))))
 
+(defvar c-fl-syn-tab-region nil)
+  ;; Non-nil when a `c-restore-string-fences' is "in force".  It's value is a
+  ;; cons of the BEG and END of the region currently "mirroring" the
+  ;; c-fl-syn-tab properties as syntax-table properties.
+
+(defun c-clear-string-fences ()
+  ;; Clear any syntax-table text properties in the region defined by
+  ;; `c-fl-syn-tab-region' which are "mirrored" by c-fl-syn-tab text
+  ;; properties.
+  (when c-fl-syn-tab-region
+    (let ((pos (car c-fl-syn-tab-region))
+	  (end (cdr c-fl-syn-tab-region)))
+      (while
+	  (and
+	   (< pos end)
+	   (setq pos
+		 (c-min-property-position pos end 'c-fl-syn-tab))
+	   (< pos end))
+	(c-clear-char-property pos 'syntax-table)
+	(setq pos (1+ pos)))
+      (setq c-fl-syn-tab-region nil))))
+
+(defun c-restore-string-fences (beg end)
+  ;; Restore any syntax-table text properties in the region (BEG END) which
+  ;; are "mirrored" by c-fl-syn-tab text properties.
+  (let ((pos beg))
+    (while
+	(and
+	 (< pos end)
+	 (setq pos
+	       (c-min-property-position pos end 'c-fl-syn-tab))
+	 (< pos end))
+      (c-put-char-property pos 'syntax-table
+			   (c-get-char-property pos 'c-fl-syn-tab))
+      (setq pos (1+ pos)))
+    (setq c-fl-syn-tab-region (cons beg end))))
+
 (defvar c-bc-changed-stringiness nil)
 ;; Non-nil when, in a before-change function, the deletion of a range of text
 ;; will change the "stringiness" of the subsequent text.  Only used when
@@ -1261,7 +1300,7 @@ c-before-change-check-unbalanced-strings
 		       "\"\\|\\s|")
 		     (point-max) t t)
 		    (progn
-		      (c-clear-char-property (1- (point)) 'syntax-table)
+		      (c-clear-syn-tab (1- (point)))
 		      (c-truncate-lit-pos-cache (1- (point)))
 		      (not (memq (char-before) c-string-delims)))))
 	       (memq (char-before) c-string-delims))
@@ -1291,10 +1330,8 @@ c-before-change-check-unbalanced-strings
 	  (cond
 	   ;; Are we escaping a newline by deleting stuff between \ and \n?
 	   ((and (> end beg)
-		 (progn
-		   (goto-char end)
-		   (eq (logand (skip-chars-backward "\\\\" beg) 1) 1)))
-	    (c-clear-char-property end 'syntax-table)
+		 (c-will-be-escaped end beg end))
+	    (c-clear-syn-tab end)
 	    (c-truncate-lit-pos-cache end)
 	    (goto-char (1+ end)))
 	   ;; Are we unescaping a newline by inserting stuff between \ and \n?
@@ -1317,15 +1354,15 @@ c-before-change-check-unbalanced-strings
 	      (let ((eoll-1 (point)))
 		(forward-char)
 		(backward-sexp)
-		(c-clear-char-property eoll-1 'syntax-table)
-		(c-clear-char-property (point) 'syntax-table)
+		(c-clear-syn-tab eoll-1)
+		(c-clear-syn-tab (point))
 		(c-truncate-lit-pos-cache (point)))
 	    ;; Opening " at EOB.
-	    (c-clear-char-property (1- (point)) 'syntax-table))
+	    (c-clear-syn-tab (1- (point))))
 	(when (and (c-search-backward-char-property 'syntax-table '(15) c-new-BEG)
 		   (memq (char-after) c-string-delims)) ; Ignore an unterminated raw string's (.
 	  ;; Opening " on last line of text (without EOL).
-	  (c-clear-char-property (point) 'syntax-table)
+	  (c-clear-syn-tab (point))
 	  (c-truncate-lit-pos-cache (point))
 	  (setq c-new-BEG (min c-new-BEG (point))))))
 
@@ -1334,7 +1371,7 @@ c-before-change-check-unbalanced-strings
 	    (and
 	     (c-search-backward-char-property 'syntax-table '(15) c-new-BEG)
 	     (memq (char-after) c-string-delims))
-	  (c-clear-char-property (point) 'syntax-table)
+	  (c-clear-syn-tab (point))
 	  (c-truncate-lit-pos-cache (point)))))
 
     (unless 
@@ -1346,13 +1383,13 @@ c-before-change-check-unbalanced-strings
 		 (not (c-characterp c-multiline-string-start-char))))
       (when (and (eq end-literal-type 'string)
 		 (not (eq (char-before (cdr end-limits)) ?\()))
-	(c-clear-char-property (1- (cdr end-limits)) 'syntax-table)
+	(c-clear-syn-tab (1- (cdr end-limits)))
 	(c-truncate-lit-pos-cache (1- (cdr end-limits)))
 	(setq c-new-END (max c-new-END (cdr end-limits))))
 
       (when (and (eq beg-literal-type 'string)
 		 (memq (char-after (car beg-limits)) c-string-delims))
-	(c-clear-char-property (car beg-limits) 'syntax-table)
+	(c-clear-syn-tab (car beg-limits))
 	(c-truncate-lit-pos-cache (car beg-limits))
 	(setq c-new-BEG (min c-new-BEG (car beg-limits)))))))
 
@@ -1375,7 +1412,7 @@ c-after-change-mark-abnormal-strings
 	   end-literal-limits end-literal-type)
 	(when (and (eq beg-literal-type 'string)
 		   (c-get-char-property (car beg-literal-limits) 'syntax-table))
-	  (c-clear-char-property (car beg-literal-limits) 'syntax-table)
+	  (c-clear-syn-tab (car beg-literal-limits))
 	  (setq c-bc-changed-stringiness (not c-bc-changed-stringiness)))
 	(setq end-literal-limits (progn (goto-char end) (c-literal-limits))
 	      end-literal-type (c-literal-type end-literal-limits))
@@ -1456,13 +1493,13 @@ c-after-change-mark-abnormal-strings
 	    (looking-at (cdr (assq (char-before) c-string-innards-re-alist))))
 	  (cond
 	   ((memq (char-after (match-end 0)) '(?\n ?\r))
-	    (c-put-char-property (1- (point)) 'syntax-table '(15))
-	    (c-put-char-property (match-end 0) 'syntax-table '(15))
+	    (c-put-syn-tab (1- (point)) '(15))
+	    (c-put-syn-tab (match-end 0) '(15))
 	    (setq c-new-BEG (min c-new-BEG (point))
 		  c-new-END (max c-new-END (match-end 0))))
 	   ((or (eq (match-end 0) (point-max))
 		(eq (char-after (match-end 0)) ?\\)) ; \ at EOB
-	    (c-put-char-property (1- (point)) 'syntax-table '(15))
+	    (c-put-syn-tab (1- (point)) '(15))
 	    (setq c-new-BEG (min c-new-BEG (point))
 		  c-new-END (max c-new-END (match-end 0))) ; Do we need c-new-END?
 	    ))
@@ -1506,16 +1543,16 @@ c-after-change-escape-NL-in-string
 				    nil t)
 		 (eq (char-after) ?\")
 		 (equal (c-get-char-property (point) 'syntax-table) '(15)))
-	(c-clear-char-property end 'syntax-table)
+	(c-clear-syn-tab end)
 	(c-truncate-lit-pos-cache end)
-	(c-clear-char-property (point) 'syntax-table)
+	(c-clear-syn-tab (point))
 	(forward-char)			; to after the "
 	(when
 	    (and
 	     ;; Search forward for an end of logical line.
 	     (re-search-forward "\\(\\\\\\(.\\|\n\\)\\|[^\\\n\r]\\)*" nil t)
 	     (memq (char-after) '(?\n ?\r)))
-	  (c-clear-char-property (point) 'syntax-table))))))
+	  (c-clear-syn-tab (point)))))))
 
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;; Parsing of quotes.
@@ -1794,6 +1831,11 @@ c-before-change
     ;; property changes.
     (when (fboundp 'syntax-ppss)
       (setq c-syntax-table-hwm most-positive-fixnum))
+;;;; NEW STOUGH, 2019-07-09
+    (unwind-protect
+	(progn
+	  (c-restore-string-fences (point-min) (point-max))
+;;;; END OF NEW STOUGH
     (save-restriction
       (save-match-data
 	(widen)
@@ -1865,7 +1907,12 @@ c-before-change
 	  )))
     ;; The following must be done here rather than in `c-after-change' because
     ;; newly inserted parens would foul up the invalidation algorithm.
-    (c-invalidate-state-cache beg)))
+    (c-invalidate-state-cache beg)
+;;;; NEW STOUGH, 2019-07-09
+    )
+      (c-clear-string-fences))
+;;;; END OF NEW STOUGH
+				  ))
 
 (defvar c-in-after-change-fontification nil)
 (make-variable-buffer-local 'c-in-after-change-fontification)
@@ -1909,6 +1956,11 @@ c-after-change
       ;; When `combine-after-change-calls' is used we might get calls
       ;; with regions outside the current narrowing.  This has been
       ;; observed in Emacs 20.7.
+;;;; NEW STOUGH, 2019-07-09
+      (unwind-protect
+	  (progn
+	    (c-restore-string-fences (point-min) (point-max))
+;;;; END OF NEW STOUGH
       (save-restriction
 	(save-match-data  ; c-recognize-<>-arglists changes match-data
 	  (widen)
@@ -1945,7 +1997,12 @@ c-after-change
 	  (save-excursion
 	    (mapc (lambda (fn)
 		    (funcall fn beg end old-len))
-		  c-before-font-lock-functions))))))
+		  c-before-font-lock-functions))))
+;;;; NEW STOUGH, 2019-07-09
+      )
+	(c-clear-string-fences))
+;;;; END OF NEW STOUGH
+						  ))
   ;; A workaround for syntax-ppss's failure to notice syntax-table text
   ;; property changes.
   (when (fboundp 'syntax-ppss)
@@ -2173,8 +2230,11 @@ c-font-lock-fontify-region
       ;; Context (etc.) fontification.
       (setq new-region (c-before-context-fl-expand-region beg end)
 	    new-beg (car new-region)  new-end (cdr new-region)))
-    (funcall (default-value 'font-lock-fontify-region-function)
-	     new-beg new-end verbose)))
+    (unwind-protect
+	(progn (c-restore-string-fences new-beg new-end)
+	       (funcall (default-value 'font-lock-fontify-region-function)
+			new-beg new-end verbose))
+      (c-clear-string-fences))))
 
 (defun c-after-font-lock-init ()
   ;; Put on `font-lock-mode-hook'.  This function ensures our after-change
@@ -2291,7 +2351,7 @@ c-electric-pair-inhibit-predicate
 invalid strings with such a syntax table text property on the
 opening \" and the next unescaped end of line."
   (if (eq char ?\")
-      (not (equal (get-text-property (1- (point)) 'syntax-table) '(15)))
+      (not (equal (get-text-property (1- (point)) 'c-fl-syn-tab) '(15)))
     (funcall (default-value 'electric-pair-inhibit-predicate) char)))
 
 \f


> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2221 bytes --]

On Tue, Jul 9, 2019 at 4:31 PM Alan Mackenzie <acm@muc.de> wrote:

>
> I'm fixing electric-pair-mode,


Hopefully that amounts to "use one of electric-pair-mode's" variables in
cc-mode.el. You can call it whatever you want.


> and I'm adding a feature to CC Mode
> specially for you, because you've asked for it so insistently.  :-)
>

I've not. I've stated what the breakage was. Really Alan, permit me
to perhaps impudently give you a piece of of Kantian advice:
don't do anything because of "someone", do it rather because
you think there is intrinsic value in it.

Still, I'm not oblivious to the fact that you have put in hours
(minutes? days?) of your life for my personal benefit. You
have my heartfelt gratitude.

Now, I've dealt with the "problem" (formerly "breakage" :-),
differently. As you know, it has existed for over a year (and I've not
"complained" since then). I've done so with a simple tweak to
a certain variable that you have created and that is "set if
the language supports multiline string literals without escaped
newlines".  I ask you again, that you bless this variable for
all users, not just me: I think you will find intrinsic value in
doing so.

It should allow you to do syntactic operations on two disjoint "s, while
> at the same time preserving CC Mode's fontification strategy.
>
> You may have less justification for complaining after trying this out.
> No apologies for that.  ;-)
>

Let me tell you how I would try this: I would apply it to all the Emacsen
I work with, in a branch. When developing C++ code for work, I would
use it and hope for good behaviour. When developing for Emacs, I would
be careful to always rebase it and not commit it. That's non-trivial work.
Now, if you tell me there may be bugs (and there might indeed be in
such a long patch) it's not a great motivation for trying it out, because
(1) these bugs would have a real impact on my work, and (2) I'm not
particularly fond of this strategy. So, I may try it, but not in the short
term. And, as was the case for more than a year, I will not "complain",
rather continue to state disagreement whenever I think it's opportune
for the common good.

João

[-- Attachment #2: Type: text/html, Size: 3366 bytes --]

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Clément Pit-Claudel]

On 2019-07-09 05:06, João Távora wrote:
> On Tue, Jul 9, 2019 at 7:42 AM Clément Pit-Claudel <cpitclaudel@gmail.com <mailto:cpitclaudel@gmail.com>> wrote:
>>
>> of the line.  That's not more "correct", of course, but it does minimize the amount
>> of refontification and blinking in a pretty common case.
> 
> No, it doesn't.

Huh? I'm claiming that:

* Single line string are a pretty common case
* Fontifying to the end of the line and not past it minimizes refontification in that case

Is that controversial?

> Suppose you had that hypothetical version of Emacs and
> you did have a multi-line string, properly escaped with backslashes. Now you
> removed the backslash. Brutal "wrong" refontification ensues just as now,
> fontifying as code what up to now used to be fontified as a string.

Yes, of course.  Did I say something else?

> Both situations are wrong, and neither is "more wrong" than the other.

I'm glad we agree. That's why I wrote 'That's not more "correct", of course'

> But the version you propose is much harder to implement, likely less
> efficient,

Strong words :) Are you saying refontifying the whole screen after the point is faster than just refontifying to the end of the line?

> Depending on how it is implemented (certainly how Alan
> implemented it) it breaks things in Emacs core and third-party code.

But surely the quality of Alan's implementation has little impact on the worthiness of the feature in general (I'm not arguing about how things should be in cc-mode today — just saying that I like the general idea)

> It is also
> useless in languages which do allow unescaped multi-line strings

And more strong words :) I'm saying it's an option I'd like to have while conceding it's not universally better, so it surely wouldn't be useless to me ^^  

Do we know what behavior is standard in other IDEs, and whether it is configurable?  On my machine I have vim, geany, gedit, and vscode, and all of the behave like Emacs currently does (but they refontify immediately, without a jit-lock idle delay).

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 3857 bytes --]

On Tue, Jul 9, 2019 at 1:33 PM Clément Pit-Claudel <cpitclaudel@gmail.com>
wrote:

> * Fontifying to the end of the line and not past it minimizes
refontification in that case
>
> Is that controversial?

I didn't say it was. I don't care about "(re)fontification minimization"
because
I don't know what that amounts to" care about:

- consistency: this is not consistent with how the rest of Emacs works.
- distraction: this is just as "distracting", and I actually like the
current "global
switch" more as it helps me spot the error easily;
- efficiency: this might well be less efficient (see below).

> > Both situations are wrong, and neither is "more wrong" than the other.
> I'm glad we agree. That's why I wrote 'That's not more "correct", of
course'

You seemed to imply there was some kind of advantage. I don't see which,
or at least I don't see any advantage that outweighs the fact that this
breaks consistency to virtually all other emacs major modes. In breaking
that consistency other minor modes and commands are broken, such
as C-M-stuff and electric-pair-mode.

> > But the version you propose is much harder to implement, likely less
> > efficient,
> Strong words :) Are you saying refontifying the whole screen after the
> point is faster than just refontifying to the end of the line?

In your model, you are not refontifying "to the end of the line".
You may have meant to say "to the next closing double quote", which may
happen after the end of the screen. While that may be less text, I would
expect perfomance of (re)fontification to vary according to the thing being
fontified. A large string is probably easier to handle than parsing
lots of other types of tokens, which is what you would be doing to
the text that was formerly a string. And there's the added complexity
to implement this to start with.

> > Depending on how it is implemented (certainly how Alan
> > implemented it) it breaks things in Emacs core and third-party code.
>
> But surely the quality of Alan's implementation has little impact on the
>  worthiness of the feature in general (I'm not arguing about how things
> should be in cc-mode today — just saying that I like the general idea)

Like I said, depends on how you value consistency to other parts of
emacs. For example, I particularly value the ease in which the
unequivocally wrong situation can be fixed. But even if you develop a
very complex implementation (presumably such as the one Alan is
developing now) which allows that, it is probably a net loss (in Emacs)
because navigation and fontification won't go hand-in-hand like before.

> > It is also
> > useless in languages which do allow unescaped multi-line strings
> And more strong words :) I'm saying it's an option I'd like to have while
conceding
> it's not universally better, so it surely wouldn't be useless to me ^^

Perhaps I'm not following. What use would it be in a language that
does allow unescaped multi-line strings? Say, what use would it
be in elisp or ruby?

> Do we know what behavior is standard in other IDEs, and whether it is
configurable?
> On my machine I have vim, geany, gedit, and vscode, and all of the behave
like Emacs
> currently does (but they refontify immediately, without a jit-lock idle
delay).

What are you doing with those other editors installed, you heretic?
Joke aside, do any of those editors have a paren-matching solution that
also
does paren-balancing? I remember searching a bit and not finding anything
interesting.

Curiously, in cc-mode, the behavior is easily configurable (earlier I
though
it didn't work but it does, it's just not officially "blessed" by the
maintainer,
and unfortunately not the default).

(add-hook 'c++-mode-hook (lambda () (setq c-multiline-string-start-char t)))

João

[-- Attachment #2: Type: text/html, Size: 5096 bytes --]

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Clément Pit-Claudel]

On 2019-07-09 10:28, João Távora wrote:
> 
> On Tue, Jul 9, 2019 at 1:33 PM Clément Pit-Claudel <cpitclaudel@gmail.com <mailto:cpitclaudel@gmail.com>> wrote:
> 
>> * Fontifying to the end of the line and not past it minimizes refontification in that case
>>
>> Is that controversial?
> 
> I didn't say it was. 

You wrote "No, it doesn't".

> I don't know what that amounts to" care about:
> 
> - consistency: this is not consistent with how the rest of Emacs works.

Which rest of Emacs? I'm saying I'd happily use an option all major modes to work the way Alan did it in cc-mode.

> - distraction: this is just as "distracting", and I actually like the current "global
> switch" more as it helps me spot the error easily;

For me there's no error to spot: in the course of typing a string, I type its opening ", then its contents, and then its closing ".  The rest of the buffer being painted in string color is distracting, and it doesn't help me spot any error.

>> > Both situations are wrong, and neither is "more wrong" than the other.
>> I'm glad we agree. That's why I wrote 'That's not more "correct", of course'
> 
> You seemed to imply there was some kind of advantage. I don't see which,

But I told you, didn't I?  I said I'd like it better because I don't like my buffers blinking :)

> or at least I don't see any advantage that outweighs the fact that this
> breaks consistency to virtually all other emacs major modes.

It would break consistency with those major modes that allow multi-line unescaped strings, indeed.  I'm not too bothered by that.

>> > But the version you propose is much harder to implement, likely less
>> > efficient,
>> Strong words :) Are you saying refontifying the whole screen after the
>> point is faster than just refontifying to the end of the line?
> 
> In your model, you are not refontifying "to the end of the line".
> You may have meant to say "to the next closing double quote", which may
> happen after the end of the screen. 

No, I meant what I say: to the end of the line.  
If I have this text:

  s = "xyz
  def main():

I prefer def main() to be highlighted as if it wasn't inside a string.  

> While that may be less text, I would
> expect perfomance of (re)fontification to vary according to the thing being
> fontified. 

In the model I'm describing, you don't have to refontify anything except the current line.  Am I missing something?

>> > It is also
>> > useless in languages which do allow unescaped multi-line strings
>> And more strong words :) I'm saying it's an option I'd like to have while conceding
>> it's not universally better, so it surely wouldn't be useless to me ^^
> 
> Perhaps I'm not following. What use would it be in a language that
> does allow unescaped multi-line strings? Say, what use would it
> be in elisp or ruby?

That's a good point.  I misunderstood your original statement.

>> Do we know what behavior is standard in other IDEs, and whether it is configurable? 
>> On my machine I have vim, geany, gedit, and vscode, and all of the behave like Emacs
>> currently does (but they refontify immediately, without a jit-lock idle delay).
> 
> What are you doing with those other editors installed, you heretic? 
> Joke aside, do any of those editors have a paren-matching solution that also
> does paren-balancing? I remember searching a bit and not finding anything
> interesting.

I'm not sure.

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 4135 bytes --]

On Tue, Jul 9, 2019 at 5:05 PM Clément Pit-Claudel <cpitclaudel@gmail.com>
wrote:
>
> On 2019-07-09 10:28, João Távora wrote:
> >
> > On Tue, Jul 9, 2019 at 1:33 PM Clément Pit-Claudel <
cpitclaudel@gmail.com <mailto:cpitclaudel@gmail.com>> wrote:
> >
> >> * Fontifying to the end of the line and not past it minimizes
refontification in that case
> >>
> >> Is that controversial?
> >
> > I didn't say it was.
>
> You wrote "No, it doesn't".

I wrote that we you said it would "minimize refontification and blinking".
I still
believe that, that it _doesn't_ minimize blinking, at least how I use
Emacs (more on that below). In your follow up you dropped the "blinking"
part, and kept the "minimize refontification" thing. And so that part is
not "controversial" (it's also not very interesting, as I tried to explain
afterwards).
>
> > I don't know what that amounts to" care about:
> >
> > - consistency: this is not consistent with how the rest of Emacs works.
>
> Which rest of Emacs? I'm saying I'd happily use an option all major modes
to work the way Alan did it in cc-mode.

But below you agree with me that it does't make much sense in
elisp or ruby.

> > - distraction: this is just as "distracting", and I actually like the
current "global
> > switch" more as it helps me spot the error easily;
>
> For me there's no error to spot: in the course of typing a string, I type
its opening ",
> then its contents, and then its closing ".  The rest of the buffer being
painted in
> string color is distracting, and it doesn't help me spot any error.

Of course. For me it never happens, _becauue I use electric-pair-mode_.
But it also probably would happen quite infrequently, if instead of
electric-pair-mode, I used another popular paren-matching package
such as smartparens, or paredit, or whatever.

So, again, it has to do with how you write code.

> >> > Both situations are wrong, and neither is "more wrong" than the
other.
> >> I'm glad we agree. That's why I wrote 'That's not more "correct", of
course'
> >
> > You seemed to imply there was some kind of advantage. I don't see which,
>
> But I told you, didn't I?  I said I'd like it better because I don't like
my buffers blinking :)

But _I_ don't see which, because I don't use Emacs the way you do.

In the next message I can start to sell the benefits of electric-pair-mode.
Oh what the heck, I can't help myself: it is designed to have you type
_exactly_ the same keys as you do now, while never unbalancing the
buffer. No blinking, really. Heck even smartparens will give you that,
in principle.

> > or at least I don't see any advantage that outweighs the fact that this
> > breaks consistency to virtually all other emacs major modes.
>
> It would break consistency with those major modes that allow
> multi-line unescaped strings, indeed.  I'm not too bothered by that.

OK. But let's not make it the default. Let consistency breakage be
a user-customized exception, not the norm.

> >> > But the version you propose is much harder to implement, likely less
> >> > efficient,
> >> Strong words :) Are you saying refontifying the whole screen after the
> >> point is faster than just refontifying to the end of the line?
> >
> > In your model, you are not refontifying "to the end of the line".
> > You may have meant to say "to the next closing double quote", which may
> > happen after the end of the screen.
>
> No, I meant what I say: to the end of the line.
> > While that may be less text, I would
> > expect perfomance of (re)fontification to vary according to the thing
being
> > fontified.
>
> In the model I'm describing, you don't have to refontify anything
> except the current line.  Am I missing something?

You and I are, I was talking about your "model" but assuming a
different usage pattern situation where you strip off a closing
backslash at the end of a multi-line string. Also I'm assuming you
are editing code, not writing code from scratch. The latter never
brings such problems, because I use a paren-matching solution.

João

[-- Attachment #2: Type: text/html, Size: 5511 bytes --]

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 467 bytes --]

On Tue, Jul 9, 2019 at 5:32 PM João Távora <joaotavora@gmail.com> wrote:

> are editing code, not writing code from scratch. The latter never
> brings such problems, because I use a paren-matching solution.

Another tweak: the former also doesn't give me such problems
also because I use such a solution, except, when I type a RET
in the middle of a string to break a long line-- and that's where
the new cc-mode bites me and the previous didn't.

João

[-- Attachment #2: Type: text/html, Size: 653 bytes --]

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Stefan Monnier]

>> Both are wrong sometimes and right at other times.
> I imagine this is less of a problem for people who use modes that
> automatically insert string delimiters (i.e. when you press ", a second
> matching quote immediately appears, so you don't run into fontification
> issues).

Depending on your usage pattern, one kind of wrong will be more frequent
than the other, yes.


        Stefan

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Alan Mackenzie]

Hello, Stefan.

On Mon, Jul 08, 2019 at 16:59:31 -0400, Stefan Monnier wrote:
> >> > Don't knock it - it's working, and will give João what he wants whilst
> >> > preserving correct fontification.
> >> Several people have already pointed out that what you call here "correct
> >> fontification" is "incorrect" in the cases where the error is a missing
> >> \ rather than a missing ".
> > "Several people" is just one that I have seen, and he is wrong.

> João gave you actual examples where fontifying the second line as
> a code rather than string was an error.  So we're at least 2.

> > Whether the error "is" a missing \ or a missing " (for whatever
> > value of "is"), the purpose of the fontification is to draw
> > attention to this error.  This error is at the EOL where there is a
> > missing " or \.  In that sense CC Mode's fontification of it is
> > correct.

> Maybe we're not talking about the same thing:

We most assuredly are.  And you still haven't read my post from Date:
Thu, 4 Jul 2019 16:42:33 +0000, which dealt in full with your next three
paragraphs.  Please go and read that post now, and either assent to its
compelling logic or take issue with it in detail.

> The addition of some kind of error/warning face on this line is indeed
> always correct and is something that can easily be done without any
> need to fiddle with syntax-table text properties.  I'm definitely not
> arguing against this highlighting of the immediate error.

> But I'm talking about is the highlighting of the subsequent line(s).

> You seem to want them to be highlighted as if the next line was not part
> of a string (i.e. assume that a " was missing on the previous line),
> whereas all other major modes highlight that second line under the
> assumption that what was missing was a final \ on the previous line (or
> whatever other syntax the corresponding language requires).

> Both are wrong sometimes and right at other times.

Not really.  The overwhelmingly most common use case is typing in a
short string which fits on one line, when the next line is (almost)
always a line of code.  It is not sensible to fontify arbitrarily large
pieces of code as a string, just because the user hasn't yet reached her
closing double quote.

It is not up to the major mode to second guess the user.  It is the
major mode's job to indicate the presence and position of an error as
clearly as possible.  CC Mode's fontification does this.  Many other
modes don't.

[ .... ]

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Stefan Monnier]

>> Both are wrong sometimes and right at other times.
> Not really.

Yes, really.  Depending on your usage pattern one will be more frequent
than the other.  Maybe in your usage pattern the second almost never
happens, but that's definitely not the case in general.

> The overwhelmingly most common use case is typing in a
> short string which fits on one line, when the next line is (almost)
> always a line of code.  It is not sensible to fontify arbitrarily large
> pieces of code as a string, just because the user hasn't yet reached her
> closing double quote.

You can't know that "the user hasn't yet reached her
closing double quote".

> It is not up to the major mode to second guess the user.

On that we agree.

> It is the major mode's job to indicate the presence and position of an
> error as clearly as possible.

The error is on the first line of the string and it's indeed good to
highlight it (and font-lock.el should probably provide support for that).

What happens with the subsequent lines is a completely different question.

> CC Mode's fontification does this.

Yes, that's good (on the first line).


        Stefan

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Alan Mackenzie]

Hello, Stefan.

On Tue, Jul 09, 2019 at 13:11:51 -0400, Stefan Monnier wrote:
> Yes, really.  .......

Until you've read, understoood, and replied to my post from Date:
Thu, 4 Jul 2019 16:42:33 +0000, this thread is just a waste of time.

Have a good afternoon.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Eli Zaretskii]

> Date: Tue, 9 Jul 2019 18:26:46 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: João Távora <joaotavora@gmail.com>,
>  emacs-devel <emacs-devel@gnu.org>
> 
> Until you've read, understoood, and replied to my post from Date:
> Thu, 4 Jul 2019 16:42:33 +0000, this thread is just a waste of time.

Alan, please stop this antisocial conduct.  If you are so annoyed, a
much better alternative is to say nothing at all.

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1715 bytes --]

On Tue, Jul 9, 2019 at 7:26 PM Alan Mackenzie <acm@muc.de> wrote:

> Not really.  The overwhelmingly most common use case is typing in a
> short string which fits on one line, when the next line is (almost)
> always a line of code.  It is not sensible to fontify arbitrarily large
> pieces of code as a string, just because the user hasn't yet reached her
> closing double quote.

You think that's the "overwhelmingly" most common use case. But if
the user is using electric-pair-mode (the thing you just "fixed", the thing
that the original author of the bug is using), it just doesn't happen.
That's
not because those users don't insert strings in their code, but because
the closing double quote is inserted for them automatically. And so no
arbitrarily large piece of code is fontified as a string _in the particular
case
of adding, say, a printf("Hello world"); to the start of a function.

Now if you don't use electric-pair-mode or another paren-matching
solution you will see the common blinking, precisely because the major
mode doesn't know if the user is keeping a closing quote in his
"mental stack". All that Stefan is saying is that you are providing for
this group of people, but that there is another group of people for which
not only the functionality you are developing isn't useful, but also
potentially harmful.

We all agree that if an unterminated string occurs, be it accidentally
by the deletion of a closing quote, or transiently (because the user
hasn't downloaded his "mental closing quote"), the matter should
be annotated on that line. There are multiple simple solutions that
do that, with no perceived drawbacks. Please consider some of them.

João

[-- Attachment #2: Type: text/html, Size: 2154 bytes --]

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Alan Mackenzie]

Hello, João.

On Tue, Jul 09, 2019 at 19:53:17 +0100, João Távora wrote:
> On Tue, Jul 9, 2019 at 7:26 PM Alan Mackenzie <acm@muc.de> wrote:

> > Not really.  The overwhelmingly most common use case is typing in a
> > short string which fits on one line, when the next line is (almost)
> > always a line of code.  It is not sensible to fontify arbitrarily large
> > pieces of code as a string, just because the user hasn't yet reached her
> > closing double quote.

> You think that's the "overwhelmingly" most common use case. But if
> the user is using electric-pair-mode (the thing you just "fixed", the thing
> that the original author of the bug is using), it just doesn't happen.

Even if electric-pair-mode is enabled, the overwhelmingly most common
use case will still be a short string which fits on a single line, and
the next line will still (almost) always be a line of code.

> That's not because those users don't insert strings in their code, but
> because the closing double quote is inserted for them automatically.
> And so no arbitrarily large piece of code is fontified as a string _in
> the particular case of adding, say, a printf("Hello world"); to the
> start of a function.

That's fine for users of electric-pair-mode, but that's a minority
sport.  Many, likely most, users don't use that mode.  For that
majority, CC Mode no longer fontifies arbitrarily large pieces of code
as a string.

> Now if you don't use electric-pair-mode or another paren-matching
> solution you will see the common blinking, precisely because the major
> mode doesn't know if the user is keeping a closing quote in his
> "mental stack".

With all due respect, I think this anthropomorphic view of major modes
supposedly "guessing" what the user wants is wide of the mark.  In C
Mode, a string starts at an opening ", and stops at the next " or
unescaped NL.  That's how a compiler handles it.  It's that simple.

> All that Stefan is saying is that you are providing for this group of
> people, but that there is another group of people for which not only
> the functionality you are developing isn't useful, but also
> potentially harmful.

I fail to see this harm.

However, there is the possibility of improving the handling of
multi-line strings in CC Mode, most notably by modifying
c-context-line-break automatically to insert backslashes, and making M-q
handle backslashes properly (which it doesn't do at the moment).

> We all agree that if an unterminated string occurs, be it accidentally
> by the deletion of a closing quote, or transiently (because the user
> hasn't downloaded his "mental closing quote"), the matter should
> be annotated on that line.

I don't agree.  Or rather, that characterization is a gross distortion
of my take on the matter, which I stated last Thursday.  This is the
passage which Stefan refuses to reply to, or even acknowledge I wrote:

>> The error is clear: There is an opening quote without a matching
>> closing quote.  The former part of the error is at the opening quote,
>> so we must indicate its position somehow, most simply by marking it
>> with warning-face.  The latter part of the error happens at the first
>> unescaped EOL; this is what defines the string as invalid.  We must
>> indicate this position as well, and the best way of doing this is
>> terminating the string-face at that position.

>> It is not arbitrary.  We are not trying to guess the intention of the
>> user; we are pointing out the objective error.


> There are multiple simple solutions that do that, with no perceived
> drawbacks. Please consider some of them.

These simple solutions all have drawbacks.  I'd already considered them,
or several of them, before arriving at the current solution for CC Mode.

What I think the real issue here is that it is difficult to adapt other
major modes to follow CC Mode's lead, since they are too tightly
constrained by the syntax-propertize-function/syntax-ppss mechanisms.

> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Lars Ingebrigtsen]

Alan Mackenzie <acm@muc.de> writes:

> That's fine for users of electric-pair-mode, but that's a minority
> sport.  Many, likely most, users don't use that mode.  For that
> majority, CC Mode no longer fontifies arbitrarily large pieces of code
> as a string.

That sounds like a good thing to me.

I understand the reasoning behind the "fontify the rest of the buffer as
a string if I've typed one double quote" thing that Emacs does in most
modes, but I don't think it's good user experience.  It's like when
you're typing in a credit card number on a web site and after you've
typed the first digit, you get a text in red saying "INVALID CREDIT CARD
NUMBER".  Hey, gimme a second...

It feels...  unfriendly...  when Emacs marks most of the buffer as
suddenly invalid when you're just trying to write a string.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1391 bytes --]

On Wed, Jul 10, 2019 at 11:41 AM Lars Ingebrigtsen <larsi@gnus.org> wrote:
>
> Alan Mackenzie <acm@muc.de> writes:
>
> > That's fine for users of electric-pair-mode, but that's a minority
> > sport.  Many, likely most, users don't use that mode.  For that
> > majority, CC Mode no longer fontifies arbitrarily large pieces of code
> > as a string.
>
> That sounds like a good thing to me.

I'd be mostly OK with the purely visual fontification change, but
if there is an accompanying change in syntax it breaks some
of the tools available to fix the situation, that is a bad thing.

> It feels...  unfriendly...  when Emacs marks most of the buffer as
> suddenly invalid when you're just trying to write a string.

Sure, and in languages such as elisp or ruby, CC-mode's approach
would be useless: you would still see the blinking. In languages
such as python and js even more complex to implement (some
strings can be multi-line and some not).

I do understand the suffering of those that don't use e-p-m. It's
one of the reasons that I started using it.  But if you don't want
to for any reason, I posted a patch to jit-lock.el in the split thread
that should  reduce if not eliminate the blinking you suffer.

I invite you (or anyone) to try it and post feedback. It will work
for any major mode, and won't introduce any breakage in
the C-M-stuff.

João

[-- Attachment #2: Type: text/html, Size: 1881 bytes --]

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Clément Pit-Claudel]

On 2019-07-10 08:24, João Távora wrote:
> I invite you (or anyone) to try it and post feedback. It will work
> for any major mode, and won't introduce any breakage in
> the C-M-stuff.

Thanks, I had a brief look.  But I'm not sure that waiting longer to blink is better: now when the buffer changes color it feels even more unexpected than the 0.5s from before.

Maybe I need to get used to it.

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 452 bytes --]

On Wed, Jul 10, 2019 at 3:15 PM Clément Pit-Claudel <cpitclaudel@gmail.com>
wrote:

>
> Thanks, I had a brief look.  But I'm not sure that waiting longer to blink
> is better:

now when the buffer changes color it feels even more unexpected than the
> 0.5s from before.
>

Hmm. Can you describe the ideal behavior in a mode such as emacs-lisp-mode?

Maybe I need to get used to it.
>

Yes, you might have Stockholm syndrome.

João

[-- Attachment #2: Type: text/html, Size: 1087 bytes --]

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Lars Ingebrigtsen]

João Távora <joaotavora@gmail.com> writes:

> I do understand the suffering of those that don't use e-p-m. It's
> one of the reasons that I started using it.  But if you don't want
> to for any reason, I posted a patch to jit-lock.el in the split thread 
> that should  reduce if not eliminate the blinking you suffer.  

It's a pretty long thread, and I couldn't find the patch.  :-/  Could
you repost?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1.1: Type: text/plain, Size: 4278 bytes --]

It's another subject split from this thread, started by Stefan, with
a few messages only. But here it is again, and a little animated gif
I made yesterday but forgot to attach.

João

diff --git a/lisp/jit-lock.el b/lisp/jit-lock.el
index 48998a81fe..8481e8ebb0 100644
--- a/lisp/jit-lock.el
+++ b/lisp/jit-lock.el
@@ -123,6 +123,15 @@ jit-lock-context-time
   :type '(number :tag "seconds")
   :group 'jit-lock)

+(defcustom jit-lock-multiline-string-grace 2
+  "Like `jit-lock-context-time' but for unterminated multiline strings.
+If the user has just opened an unterminated string at EOL, give
+him/her some grace time before deciding it is a multi-line string
+and fontifying accordingly, do so only if the user stares idle at
+that string for more than this many seconds."
+  :type '(number :tag "seconds")
+  :group 'jit-lock)
+
 (defcustom jit-lock-defer-time nil ;; 0.25
   "Idle time after which deferred fontification should take place.
 If nil, fontification is not deferred.
@@ -232,7 +241,7 @@ jit-lock-mode
       (unless jit-lock-context-timer
         (setq jit-lock-context-timer
               (run-with-idle-timer jit-lock-context-time t
-                                   'jit-lock-context-fontify)))
+                                   (jit--lock-context-timer-function))))
       (setq jit-lock-context-unfontify-pos
             (or jit-lock-context-unfontify-pos (point-max))))

@@ -306,6 +315,44 @@ jit-lock--debug-fontify
                                    pos 'fontified)))))))))
       (setq jit-lock-defer-buffers nil))))

+(defun jit--lock-context-timer-function ()
+  (let (last        ; point marker the last time context timer was run
+        in-s-or-c-p ; t if in string or comment that time around
+        grace-timer ; idle timer for fontifying unterminated s-or-c, or nil
+        )
+    (lambda ()
+      (let ((point (point-marker))
+            (new-in-s-or-c-p
+             (nth 8 (save-excursion (syntax-ppss (line-end-position))))))
+        (if (and jit-lock-multiline-string-grace
+                 last
+                 (eq (marker-buffer last) (current-buffer))
+                 (eq (line-number-at-pos last) (line-number-at-pos)))
+            (cond ((and (null in-s-or-c-p) new-in-s-or-c-p (null
grace-timer))
+                   (setq grace-timer
+                         (run-with-idle-timer
jit-lock-multiline-string-grace nil
+                                              (lambda ()
+                                                (jit-lock-context-fontify)
+                                                (setq grace-timer nil)))))
+                  ((and in-s-or-c-p
+                        (null new-in-s-or-c-p)
+                        grace-timer)
+                   (cancel-timer grace-timer)
+                   (setq grace-timer nil))
+                  (t
+                   ;; no state change, leave everything as it was
+                   ))
+          ;; left the line somehow or customized feature away: cancel
+          ;; everything, resume normal operation.
+          (when grace-timer
+            (cancel-timer grace-timer)
+            (setq grace-timer nil)))
+        ;; proceed as usual, unless grace-timer is counting
+        (unless grace-timer
+          (jit-lock-context-fontify))
+        (setq last point in-s-or-c-p new-in-s-or-c-p)))))
+
+
 (defun jit-lock-register (fun &optional contextual)
   "Register FUN as a fontification function to be called in this buffer.
 FUN will be called with two arguments START and END indicating the region

On Thu, Jul 11, 2019 at 4:15 PM Lars Ingebrigtsen <larsi@gnus.org> wrote:

> João Távora <joaotavora@gmail.com> writes:
>
> > I do understand the suffering of those that don't use e-p-m. It's
> > one of the reasons that I started using it.  But if you don't want
> > to for any reason, I posted a patch to jit-lock.el in the split thread
> > that should  reduce if not eliminate the blinking you suffer.
>
> It's a pretty long thread, and I couldn't find the patch.  :-/  Could
> you repost?
>
> --
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no
>


-- 
João Távora

[-- Attachment #1.2: Type: text/html, Size: 5501 bytes --]

[-- Attachment #2: blinking-no-blinking.gif --]
[-- Type: image/gif, Size: 192332 bytes --]

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Lars Ingebrigtsen]

João Távora <joaotavora@gmail.com> writes:

> It's another subject split from this thread, started by Stefan, with
> a few messages only. But here it is again, and a little animated gif
> I made yesterday but forgot to attach.

Hm...  I'm getting "malformed patch"...

> +  "Like `jit-lock-context-time' but for unterminated multiline strings.
> +If the user has just opened an unterminated string at EOL, give
> +him/her some grace time before deciding it is a multi-line string
> +and fontifying accordingly, do so only if the user stares idle at
> +that string for more than this many seconds."

I think this is a good idea.  A slight tweak to this would be to do the
fontifying immediately if the user moves off of the line with the
unterminated string, too.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 4770 bytes --]

On Thu, Jul 11, 2019 at 4:51 PM Lars Ingebrigtsen <larsi@gnus.org> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > It's another subject split from this thread, started by Stefan, with
> > a few messages only. But here it is again, and a little animated gif
> > I made yesterday but forgot to attach.
>
> Hm...  I'm getting "malformed patch"...

Some Gmail messup, maybe (my Gnus configuration is broken in
sometimes).

I try again at the end of this mail, after having asked gmail
to send unformatted text.

> > +  "Like `jit-lock-context-time' but for unterminated multiline strings.
> > +If the user has just opened an unterminated string at EOL, give
> > +him/her some grace time before deciding it is a multi-line string
> > +and fontifying accordingly, do so only if the user stares idle at
> > +that string for more than this many seconds."
>
> I think this is a good idea.  A slight tweak to this would be to do the
> fontifying immediately if the user moves off of the line with the
> unterminated string, too.

You mean after the system has discovered that he/she has recently
created an string? If so, it makes sense. Otherwise we would be
context-fontifying more frequently than we did before, and that
could create a performance problem.

I'll add this to the TODO list.

diff --git a/lisp/jit-lock.el b/lisp/jit-lock.el
index 48998a81fe..8481e8ebb0 100644
--- a/lisp/jit-lock.el
+++ b/lisp/jit-lock.el
@@ -123,6 +123,15 @@ jit-lock-context-time
   :type '(number :tag "seconds")
   :group 'jit-lock)

+(defcustom jit-lock-multiline-string-grace 2
+  "Like `jit-lock-context-time' but for unterminated multiline strings.
+If the user has just opened an unterminated string at EOL, give
+him/her some grace time before deciding it is a multi-line string
+and fontifying accordingly, do so only if the user stares idle at
+that string for more than this many seconds."
+  :type '(number :tag "seconds")
+  :group 'jit-lock)
+
 (defcustom jit-lock-defer-time nil ;; 0.25
   "Idle time after which deferred fontification should take place.
 If nil, fontification is not deferred.
@@ -232,7 +241,7 @@ jit-lock-mode
       (unless jit-lock-context-timer
         (setq jit-lock-context-timer
               (run-with-idle-timer jit-lock-context-time t
-                                   'jit-lock-context-fontify)))
+                                   (jit--lock-context-timer-function))))
       (setq jit-lock-context-unfontify-pos
             (or jit-lock-context-unfontify-pos (point-max))))

@@ -306,6 +315,44 @@ jit-lock--debug-fontify
                                    pos 'fontified)))))))))
       (setq jit-lock-defer-buffers nil))))

+(defun jit--lock-context-timer-function ()
+  (let (last        ; point marker the last time context timer was run
+        in-s-or-c-p ; t if in string or comment that time around
+        grace-timer ; idle timer for fontifying unterminated s-or-c, or nil
+        )
+    (lambda ()
+      (let ((point (point-marker))
+            (new-in-s-or-c-p
+             (nth 8 (save-excursion (syntax-ppss (line-end-position))))))
+        (if (and jit-lock-multiline-string-grace
+                 last
+                 (eq (marker-buffer last) (current-buffer))
+                 (eq (line-number-at-pos last) (line-number-at-pos)))
+            (cond ((and (null in-s-or-c-p) new-in-s-or-c-p (null
grace-timer))
+                   (setq grace-timer
+                         (run-with-idle-timer
jit-lock-multiline-string-grace nil
+                                              (lambda ()
+                                                (jit-lock-context-fontify)
+                                                (setq grace-timer nil)))))
+                  ((and in-s-or-c-p
+                        (null new-in-s-or-c-p)
+                        grace-timer)
+                   (cancel-timer grace-timer)
+                   (setq grace-timer nil))
+                  (t
+                   ;; no state change, leave everything as it was
+                   ))
+          ;; left the line somehow or customized feature away: cancel
+          ;; everything, resume normal operation.
+          (when grace-timer
+            (cancel-timer grace-timer)
+            (setq grace-timer nil)))
+        ;; proceed as usual, unless grace-timer is counting
+        (unless grace-timer
+          (jit-lock-context-fontify))
+        (setq last point in-s-or-c-p new-in-s-or-c-p)))))
+
+
 (defun jit-lock-register (fun &optional contextual)
   "Register FUN as a fontification function to be called in this buffer.
 FUN will be called with two arguments START and END indicating the region

[-- Attachment #2: Type: text/html, Size: 5827 bytes --]

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Lars Ingebrigtsen]

João Távora <joaotavora@gmail.com> writes:

>> Hm...  I'm getting "malformed patch"...
>
> Some Gmail messup, maybe (my Gnus configuration is broken in 
> sometimes).
>
> I try again at the end of this mail, after having asked gmail
> to send unformatted text.

Still getting "malformed"; I think with Gmail the only way to post
patches is to include the patch in an attachment.

>> I think this is a good idea.  A slight tweak to this would be to do the
>> fontifying immediately if the user moves off of the line with the
>> unterminated string, too.
>
> You mean after the system has discovered that he/she has recently
> created an string?

Yes.  Or if the system has identified that an unterminated string
exists, and the user moves to a different line outside the
fontified-as-unterminated-string area.  If that's possible.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 872 bytes --]

On Fri, Jul 12, 2019 at 4:58 PM Lars Ingebrigtsen <larsi@gnus.org> wrote:

> Still getting "malformed"; I think with Gmail the only way to post
> patches is to include the patch in an attachment.

Probably. I normally copy-paste the thing into a buffer and
M-x diff-mode  there, fix any bugs manually.

> Yes.  Or if the system has identified that an unterminated string
> exists, and the user moves to a different line outside the
> fontified-as-unterminated-string area.  If that's possible.  :-)

It is, I think.

So the feature, which I have dubbed "the antiblink"  is now
in a scratch/jit-lock-antiblink branch. It should be much more stable
and works mostly from the post-command-hook and a new idle timer.
It includes your "immediate refontify if left" idea, Lars.  And Stefan,
the code should now be much easier to understand.

Cheers,
João

[-- Attachment #2: Type: text/html, Size: 1128 bytes --]

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Lars Ingebrigtsen]

João Távora <joaotavora@gmail.com> writes:

> So the feature, which I have dubbed "the antiblink"  is now 
> in a scratch/jit-lock-antiblink branch. It should be much more stable
> and works mostly from the post-command-hook and a new idle timer.
> It includes your "immediate refontify if left" idea, Lars.  And Stefan,
> the code should now be much easier to understand.

I tried it for a few minutes, and it seems a lot nicer now to add
strings to code.  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 5196 bytes --]

On Wed, Jul 10, 2019 at 11:32 AM Alan Mackenzie <acm@muc.de> wrote:

> Hello, João.
>
> On Tue, Jul 09, 2019 at 19:53:17 +0100, João Távora wrote:
> > On Tue, Jul 9, 2019 at 7:26 PM Alan Mackenzie <acm@muc.de> wrote:
>
> > > Not really.  The overwhelmingly most common use case is typing in a
> > > short string which fits on one line, when the next line is (almost)
> > > always a line of code.  It is not sensible to fontify arbitrarily large
> > > pieces of code as a string, just because the user hasn't yet reached
> her
> > > closing double quote.
>
> > You think that's the "overwhelmingly" most common use case. But if
> > the user is using electric-pair-mode (the thing you just "fixed", the
> thing
> > that the original author of the bug is using), it just doesn't happen.
>
> Even if electric-pair-mode is enabled, the overwhelmingly most common
> use case will still be a short string which fits on a single line, and
> the next line will still (almost) always be a line of code.
>

You misunderstood me. What I said "just doens't happen" is "fontify
arbitrarily
large pieces".

That's fine for users of electric-pair-mode, but that's a minority
> sport.  Many, likely most, users don't use that mode.  For that
> majority, CC Mode no longer fontifies arbitrarily large pieces of code
> as a string.
>

Even if electric-pair-mode users are a minority users of autopairing
solutions such as smartparens are most definitely not. Every modern
editor has one of those.  The very same applies to those users. But even
if those users are a minority (< 50%), it still doesn't justify breaking
long-held behavior that they expect.


> > Now if you don't use electric-pair-mode or another paren-matching
> > solution you will see the common blinking, precisely because the major
> > mode doesn't know if the user is keeping a closing quote in his
> > "mental stack".
>
> With all due respect, I think this anthropomorphic view of major modes
> supposedly "guessing" what the user wants is wide of the mark.


Precisely. I say it _doesn't_ know and _can't_ know. So any guess,
such as the one you're taking, that it's an invalid string, is wrong
some percentage of the time.


> In C
> Mode, a string starts at an opening ", and stops at the next " or
> unescaped NL.  That's how a compiler handles it.  It's that simple.
>

The compiler doesn't edit code. It's irrelevant how it views the lines after
the error, so long as it hightlights the error in the correct line.


> > All that Stefan is saying is that you are providing for this group of
> > people, but that there is another group of people for which not only
> > the functionality you are developing isn't useful, but also
> > potentially harmful.
>
> I fail to see this harm.
>

Stepping carefully on the terms, you have made two patches since
this "problem" happened:

1. One to restore the previous e-p-m behaviour in cc-mode
2. A long patch devised "just for me" to restore previous C-M-behaviour

The necessity of such patches, and the fact that patch 2 is likely buggy,
according to you, can be seen, in my opinion, as an expression of "harm".

> We all agree that if an unterminated string occurs, be it accidentally
> > by the deletion of a closing quote, or transiently (because the user
> > hasn't downloaded his "mental closing quote"), the matter should
> > be annotated on that line.
>
> I don't agree.  Or rather, that characterization is a gross distortion
> of my take on the matter, which I stated last Thursday.  This is the
> passage which Stefan refuses to reply to, or even acknowledge I wrote:
>
> >> The error is clear: There is an opening quote without a matching
> >> closing quote.  The former part of the error is at the opening quote,
> >> so we must indicate its position somehow, most simply by marking it
> >> with warning-face.  The latter part of the error happens at the first
> >> unescaped EOL; this is what defines the string as invalid.  We must
> >> indicate this position as well, and the best way of doing this is
> >> terminating the string-face at that position.
>
> >> It is not arbitrary.  We are not trying to guess the intention of the
> >> user; we are pointing out the objective error.
>

So the "gross distortion" is that you think it's absolutely necessary that
the error be annotated both at string start and invalid string end? Only one
of them is are totally wrong for you. Is that it, the thursday thing?

Doesn't seem so important for me, but if it were, the flymake backend
I proposed can highlight wherever you most desire. I don't know about
Stefan's solution, though.

Does the compiler tell the user about both locations?


> > There are multiple simple solutions that do that, with no perceived
> > drawbacks. Please consider some of them.
>
> These simple solutions all have drawbacks.  I'd already considered them,
> or several of them, before arriving at the current solution for CC Mode.
>

So can you summarize, for my benefit:

- What are the drawbacks of Stefan's solution?
- What are the drawbacks of my flymake-based soluion?

João

[-- Attachment #2: Type: text/html, Size: 7423 bytes --]

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Alan Mackenzie]

Hello, João.

On Wed, Jul 10, 2019 at 13:10:45 +0100, João Távora wrote:
> On Wed, Jul 10, 2019 at 11:32 AM Alan Mackenzie <acm@muc.de> wrote:

> > On Tue, Jul 09, 2019 at 19:53:17 +0100, João Távora wrote:
> > > On Tue, Jul 9, 2019 at 7:26 PM Alan Mackenzie <acm@muc.de> wrote:

> > > > Not really.  The overwhelmingly most common use case is typing in a
> > > > short string which fits on one line, when the next line is (almost)
> > > > always a line of code.  It is not sensible to fontify arbitrarily large
> > > > pieces of code as a string, just because the user hasn't yet reached
> > > > her closing double quote.

> > > You think that's the "overwhelmingly" most common use case. But if
> > > the user is using electric-pair-mode (the thing you just "fixed", the
> > > thing that the original author of the bug is using), it just
> > > doesn't happen.

> > Even if electric-pair-mode is enabled, the overwhelmingly most common
> > use case will still be a short string which fits on a single line, and
> > the next line will still (almost) always be a line of code.

> You misunderstood me. What I said "just doens't happen" is "fontify
> arbitrarily large pieces".

OK.  Sorry about that.

> > That's fine for users of electric-pair-mode, but that's a minority
> > sport.  Many, likely most, users don't use that mode.  For that
> > majority, CC Mode no longer fontifies arbitrarily large pieces of
> > code as a string.

> Even if electric-pair-mode users are a minority users of autopairing
> solutions such as smartparens are most definitely not. Every modern
> editor has one of those.

I'll believe you, though I don't know what you mean by smartparens.

> The very same applies to those users. But even if those users are a
> minority (< 50%), it still doesn't justify breaking long-held behavior
> that they expect.

Or are we talking about fixing long standing breakage they've had to
tolerate?  At least two people on this list have said they welcome the CC
Mode style of fontification of invalid strings.

> > > Now if you don't use electric-pair-mode or another paren-matching
> > > solution you will see the common blinking, precisely because the
> > > major mode doesn't know if the user is keeping a closing quote in
> > > his "mental stack".

> > With all due respect, I think this anthropomorphic view of major modes
> > supposedly "guessing" what the user wants is wide of the mark.

> Precisely. I say it _doesn't_ know and _can't_ know. So any guess, such
> as the one you're taking, that it's an invalid string, is wrong some
> percentage of the time.

Neither CC Mode nor I guess.  "We" both use a definition of "string" that
obviates guessing.  (See next paragraph.)

> > In C Mode, a string starts at an opening ", and stops at the next "
> > or unescaped NL.  That's how a compiler handles it.  It's that
> > simple.

> The compiler doesn't edit code. It's irrelevant how it views the lines
> after the error, so long as it hightlights the error in the correct
> line.

What's that got to do with it?  If you don't like CC Mode's definition of
a string, propose an alternative here, a non-vague alternative, and we
can discuss it.

> > > All that Stefan is saying is that you are providing for this group
> > > of people, but that there is another group of people for which not
> > > only the functionality you are developing isn't useful, but also
> > > potentially harmful.

> > I fail to see this harm.

> Stepping carefully on the terms, you have made two patches since
> this "problem" happened:

> 1. One to restore the previous e-p-m behaviour in cc-mode
> 2. A long patch devised "just for me" to restore previous C-M-behaviour

Yes.  There were bugs in the interface between CC Mode and
electric-pair-mode, and half of us had to sort these out.  It's a good
job that one of us was able to step up, and diagnose and fix this.

As for 2, what you want there, as I keep saying, is bogus.  You want to
treat two syntactically disjoint "s as though they enclosed a string.
Although this is bogus, I have some sympathy with people who want to do
this, on the grounds that it "worked" in the past.  But it is
fundamentally bogus.

> The necessity of such patches, and the fact that patch 2 is likely buggy,
> according to you, can be seen, in my opinion, as an expression of "harm".

Must you be like that?  That patch is not "likely buggy", any more than
any other patch.  There is one particular bug in it, a very obscure
situation, where a raw string's identifier which contains "s gets
confused with another raw string when the defining delimiters are damaged
in a particular fashion.  Not something you're liable to stumble over,
but my integrity prevents me from representing the patch as bug free.  It
will be getting fixed nevertheless.

As you acknowleged in another post, that patch took some hours to write.
Would you please, as the person clamouring for it, try it out now and
give me appropriate feedback.  I would rather get that feedback before
committing it to savannah than afterwards.

> > > We all agree that if an unterminated string occurs, be it
> > > accidentally by the deletion of a closing quote, or transiently
> > > (because the user hasn't downloaded his "mental closing quote"),
> > > the matter should be annotated on that line.

> > I don't agree.  Or rather, that characterization is a gross
> > distortion of my take on the matter, which I stated last Thursday.
> > This is the passage which Stefan refuses to reply to, or even
> > acknowledge I wrote:

> > >> The error is clear: There is an opening quote without a matching
> > >> closing quote.  The former part of the error is at the opening quote,
> > >> so we must indicate its position somehow, most simply by marking it
> > >> with warning-face.  The latter part of the error happens at the first
> > >> unescaped EOL; this is what defines the string as invalid.  We must
> > >> indicate this position as well, and the best way of doing this is
> > >> terminating the string-face at that position.

> > >> It is not arbitrary.  We are not trying to guess the intention of the
> > >> user; we are pointing out the objective error.


> So the "gross distortion" is that you think it's absolutely necessary that
> the error be annotated both at string start and invalid string end?

Yes.

> Only one of them is are totally wrong for you. Is that it, the thursday
> thing?

Sorry, I can't parse that.

> Doesn't seem so important for me, but if it were, the flymake backend
> I proposed can highlight wherever you most desire. I don't know about
> Stefan's solution, though.

> Does the compiler tell the user about both locations?

I don't know.  With having the correct highlighting in CC Mode, invalid
strings get corrected before being submitted to the compiler.  ;-)

> > > There are multiple simple solutions that do that, with no perceived
> > > drawbacks. Please consider some of them.

> > These simple solutions all have drawbacks.  I'd already considered
> > them, or several of them, before arriving at the current solution for
> > CC Mode.


> So can you summarize, for my benefit:

> - What are the drawbacks of Stefan's solution?

It doesn't, at least without an awful lot of effort, fontify an invalid
string in the correct CC Mode fashion.  It would leave an arbitrary
amount of text after the end of an invalid string fontified with
string-face.

> - What are the drawbacks of my flymake-based soluion?

It's not part of CC Mode.  It would involve loading another package, and
likely would be slower than CC Mode's fontification.  It would not be
well integrated with CC Mode, and would possibly need a lot of work to
make it work properly.

> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 5462 bytes --]

On Wed, Jul 10, 2019 at 3:03 PM Alan Mackenzie <acm@muc.de> wrote:
> I'll believe you, though I don't know what you mean by smartparens.

See https://github.com/Fuco1/smartparens.

> Or are we talking about fixing long standing breakage they've had to
> tolerate?  At least two people on this list have said they welcome the CC
> Mode style of fontification of invalid strings.

I wasn't talking about that, I was talking about the cc/e-p-m related bug
that a user reported, and the C-M-stuff that I use.  This is a behavior
in emacs that probably existed since cc-mode was first created up
to last year.

But, we can start talking about that "tolerated breakage", by
which you probably mean what has been recently described here as
"blinking" a lower portion of the screen between string/non-string
fontification for people that don't use a delimiter-pairing solution.
And we could start by examining solutions that:

1. Don't have any drawbacks to e-p-m and C-M-navigation;
2. Fix it in all of Emacs, even in modes that _do_ have multi-line strings;

We've established that your cc-mode solution does not verify 1 (and by
definition it does not verify 2).

> What's that got to do with it?  If you don't like CC Mode's definition of
> a string, propose an alternative here, a non-vague alternative, and we
> can discuss it.

Very well: I want the previous CC Mode's definition of a string, before your
changes of June 2018. This is so non-vague that it's even coded up in
the git repository. At the very least  I want to be able to switch to
it, and also think it should be the default, because it's (quite probably)
the definition that has existed from cc-mode's inception in 19xx up to
June 2018.

> Yes.  There were bugs in the interface between CC Mode and
> electric-pair-mode, and half of us had to sort these out.  It's a good
> job that one of us was able to step up, and diagnose and fix this.

That's a heartwarming comment, because it's always heartwarming
that the person who fixes it was the one that broke it in the first
place. Pardon, "committed actions that made it work differently than
it did before, causing a bug report".

> As for 2, what you want there, as I keep saying, is bogus.  You want to
> treat two syntactically disjoint "s as though they enclosed a string.
> Although this is bogus, I have some sympathy with people who want to do
> this, on the grounds that it "worked" in the past.  But it is
> fundamentally bogus.

Two "syntactically disjoint" double quotes in C are _always_ bogus. There
can be 0 such occurrences in a correct C program. The only job the compiler
has is to not compile the program and presumably signal the error. The
editor should do so, too. It _doesnt_ need to do it by implementing
exactly the syntax of the C standard, because the editor is not not
a compiler.

And that's the sensible approach that Emacs has taken since a
long time.

Again, you may argue that with it comes a "blinking" problem for those
that don't use quote-pairing solutions. I will agree. And that problem
happens with comments, too, and in many many other modes. It's not
exclusive to cc-mode.

> Must you be like that?  That patch is not "likely buggy", any more than
> any other patch.

You said "the enhancement is not yet bug free". That makes it a
least "likely buggy".

> There is one particular bug in it, a very obscure

Oh, then I think that makes it 100% surely buggy!

> As you acknowleged in another post, that patch took some hours to write.
> Would you please, as the person clamouring for it,

I thanked you for your great unsolicited efforts, but now I must solicit
a final one: where exactly did I "clamour" for it? I told you from the
beginning that the problem doesn't affect me in particular. I also don't
remember having encouraged you to persist in that particular
solution for other users sake.

Indeed I hope it is never installed, because frankly, breaking
the no-disparagement rule, it looks ghastly. Seriously, look
at it.

> > - What are the drawbacks of Stefan's solution?
> It doesn't, at least without an awful lot of effort, fontify an invalid
> string in the correct CC Mode fashion.  It would leave an arbitrary
> amount of text after the end of an invalid string fontified with
> string-face.

OK. Since I don't know Stefan's solution, I won't refute. But I will
ask:

1. What is "the correct CC Mode fashion" these days?
2. What is the problem with the second part. Is it the "blinking"
problem that I agreed to earlier? Or is it something else? Are you
preoccupied that the syntax of an invalid program as shown in
cc-mode is exactly like some compiler's internal view of it?

> > - What are the drawbacks of my flymake-based soluion?
> It's not part of CC Mode.  It would involve loading another package, and
> likely would be slower than CC Mode's fontification.  It would not be
> well integrated with CC Mode, and would possibly need a lot of work to
> make it work properly.

It already works properly. In many cases it does not need to be "integrated"
with the major mode at all. Also probably faster, depending on whether you
launch the compiler process asynchronously (which is the default).  I'd say
you have to try it. It doesn't need to be flymake, you can ask folks about
flycheck, too, which many people prefer (https://www.flycheck.org/en/latest/
).

João

[-- Attachment #2: Type: text/html, Size: 6866 bytes --]

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Alan Mackenzie]

Hello, João.

On Wed, Jul 10, 2019 at 17:05:20 +0100, João Távora wrote:

[ .... ]

I'm withdrawing from this conversation.  My feeling at the moment is
that of having been pummelled day after day by wholly negative and
destructive criticism of my work.  I don't need any more of this at the
moment.

Just for the record, CC Mode's fontification is NOT going back to the
inferior state it had up until about a year ago.  Anything else is up
for negotiation.

> João

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [PATCH] Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I'm withdrawing from this conversation.  My feeling at the moment is
  > that of having been pummelled day after day by wholly negative and
  > destructive criticism of my work.  I don't need any more of this at the
  > moment.

I am very sorry that our conversation caused such bad feelings.
Can we all think about how we could have led it to a better outcome?

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Stefan Monnier]

> I think, perhaps, you encounter such broken strings extremely rarely,
> and you're making a big song and dance over a relatively minor matter.

This argument cuts both ways.


        Stefan

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Stefan Monnier]

> evasive and answer the point: can you suggest some other mechanism by
> which CC Mode can create the required font-locking?

Should be pretty easy: add a rule to font-lock-keywords (i.e. no change
in syntax-tables) matching a double quote and which then checks with
syntax-ppss if it's an opening or closing quote and whether the matching
matching quote is on the same line or not (and applies a warning face
accordingly).


        Stefan

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Alan Mackenzie]

Hello, Stefan.

On Wed, Jul 03, 2019 at 12:58:32 -0400, Stefan Monnier wrote:
> > evasive and answer the point: can you suggest some other mechanism by
> > which CC Mode can create the required font-locking?

> Should be pretty easy: add a rule to font-lock-keywords (i.e. no change
> in syntax-tables) matching a double quote and which then checks with
> syntax-ppss if it's an opening or closing quote and whether the matching
> matching quote is on the same line or not (and applies a warning face
> accordingly).

Thanks.  That's part of the problem.  Another part is that syntax
fontification will have wrongly fontified the bits beyond the end of the
invalid string with font-lock-string-face.

I've had another idea - set these syntax-table properties as is done
currently, but only for the duration of font-lock-fontify-region.  That
way, the fontification should come out right, and João will be able to
do the things he wants to do with invalid strings.

I would do this in CC Mode by setting some text property instead of
syntax-table (let's call it c-fl-syn-tab), and just before font locking
happens, copying the value of these properties to the syntax-table
property.  And, of course, the reverse process after the font locking.
Or something like that.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Stefan Monnier]

Hi Alan,

> Thanks.  That's part of the problem.  Another part is that syntax
> fontification will have wrongly fontified the bits beyond the end of the
> invalid string with font-lock-string-face.

We've gone through that several times already.
We don't and can't know if it's wrong or not, because it depends on what
is the error: is there a missing closing " on the line, or should the
newline be preceded by \ ?

So no matter which behavior we choose to use, it will sometimes be
wrong, and the frequency of which is right and which is wrong completely
depends on things like the age of the captain.

You decided to expend a lot of efforts on implementing
a differently-wrong behavior than the default wrong behavior, breaking
long standing useful wrong behavior along the way.  That's your
prerogative.  But it's still wrong anyway because there can be no right
behavior here.


        Stefan

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Alan Mackenzie]

Hello, Stefan.

On Thu, Jul 04, 2019 at 11:52:32 -0400, Stefan Monnier wrote:
> Hi Alan,

> > Thanks.  That's part of the problem.  Another part is that syntax
> > fontification will have wrongly fontified the bits beyond the end of the
> > invalid string with font-lock-string-face.

> We've gone through that several times already.
> We don't and can't know if it's wrong or not, because it depends on what
> is the error: is there a missing closing " on the line, or should the
> newline be preceded by \ ?

The error is clear: There is an opening quote without a matching closing
quote.  The former part of the error is at the opening quote, so we must
indicate its position somehow, most simply by marking it with
warning-face.  The latter part of the error happens at the first
unescaped EOL; this is what defines the string as invalid.  We must
indicate this position as well, and the best way of doing this is
terminating the string-face at that position.

It is not arbitrary.  We are not trying to guess the intention of the
user; we are pointing out the objective error.

[ .... ]

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[Stefan Monnier]

>> > Thanks.  That's part of the problem.  Another part is that syntax
>> > fontification will have wrongly fontified the bits beyond the end of the
>> > invalid string with font-lock-string-face.
>
>> We've gone through that several times already.
>> We don't and can't know if it's wrong or not, because it depends on what
>> is the error: is there a missing closing " on the line, or should the
>> newline be preceded by \ ?
>
> The error is clear: [...]

On the first line, yes.  But in the above, we're discussing "the bits
beyond the end of the invalid string".


        Stefan

Re: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 314 bytes --]

On Thu, Jul 4, 2019 at 5:43 PM Alan Mackenzie <acm@muc.de> wrote:
>  we are pointing out the objective error.

Sure, and all of the near handful of solutions that are on the table
do exactly that. The difference is that the one you came up with
breaks other parts of Emacs that needn't be affected.

João

[-- Attachment #2: Type: text/html, Size: 418 bytes --]

bug#36474: Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 3741 bytes --]

On Tue, Jul 2, 2019 at 5:04 PM Alan Mackenzie <acm@muc.de> wrote:

> > [did you mean to copy bug-gnu-emacs@gnu.org, or emacs-devel@gnu.org?
> > I'm assuming the latter and correcting]
> No, it's a bug, therefore I submitted a bug report.

You should use the X-Debbugs-CC feature then (and why not continue
in the existing bug 36423?)

Anyway, I insist this matter be brought to emacs-devel because it's a
followup to a discussion that started there but never reached a
suitable conclusion. For that reason, and because I provide a
workaround for the bug  at the end of  this message, I'm cross-posting
this one mail to both the bug list and emacs-devel.

> > The arguments _against_ NL-terminated strings is that they (1) break
> > longstanding features such as sexp-based navigation (e.g. `up-list`
> > and friends) for modes such say, `js-mode` and (2) break features
> > that expect this to work, most notably electric-pair-mode.
>
> This isn't true.

What "isn't true"? Have those features broken or not? They worked
before the fe06f643b commit and don't work after the commit. It
sounds quite unrefutable to me.

> If those other feature no longer work with an up to
> date Emacs, they should be fixed.

I've stated this repeatedly in the life of this discussion: it's not just
about
electric-pair-mode. If you try to M-x up-list from a multi-line string in
emacs 26.1 it works just as well in js-mode and c++-mode.  In emacs
master it does not in c++-mode. Same with forward-sexp on the starting
delimiter, etc.

> The fontification that CC Mode does is natural and helpful, and users
> haven't complained about it (except when there've been bugs).

Yes, users haven't complained except when users have complained.

> There have
> certainly been no complaints about using font-lock-warning-face for the
> invalid string delimiters, and font-lock-string-face for valid ones.

That's because providing this annotation is perfectly fine.  The problem
is providing it _at the expense of other features_. And _that's_ what
they've complained about: an average user has no obvious way of
telling that the particular implementation of the red annotation thingy
is guilty of breaking his C-M-u or his electric-pair-mode.

He/she might even judge the latter more vital than said red thingy
, an annotation which he/she will get by other means if using
very popular packages such as flycheck, or flymake, or eglot, or
lsp-mode, etc. These normally call the compiler directly on the
source code and highlight those and many other errors.

On the other hand, if what you want is the red annotation, are you
absolutely sure there isn't a better way to get it? And if you are,
are you also absolutely sure you need to put it in the code and
and not provide an easy way to turn it off?

> For this, I think we would both rather that you amend elec-pair.el rather
> than me.

I'll be "mulling this over". There are potentially many other points of
breakage that would need such an indirection, and doing that to serve
just a particular cc-mode quirk doesn't sound priority to me.

In the meantime, let others chime in.

Also, in the meantime, for a user that is bothered by this bug,
I'd recomend to put something like this in his/her .emacs file:

  (defun c-unescaped-nls-in-string-p (&optional quote-pos) t)

I had something more elaborate in my setup but just this
seems to fix it in my testing.

There is a also a very promising variable, c-multiline-string-start-char,
that I think would be a good candidate for customizing this, but I
haven't messed with it enough. Just setting it from .emacs doesn't
do the trick. Perhaps in a mode hook?

--
João Távora

[-- Attachment #2: Type: text/html, Size: 5107 bytes --]

bug#36474: Acknowledgement (Algorithm in electric-pair--unbalanced-strings-p unsuitable for CC Mode)

[Alan Mackenzie]

The bug has been fixed in CC Mode.

On Tue, Jul 02, 2019 at 13:17:01 +0000, GNU bug Tracking System wrote:
> Thank you for filing a new bug report with debbugs.gnu.org.

> This is an automatically generated reply to let you know your message
> has been received.

> Your message is being forwarded to the package maintainers and other
> interested parties for their attention; they will reply in due course.

> Your message has been sent to the package maintainer(s):
>  bug-gnu-emacs@gnu.org

> If you wish to submit further information on this problem, please
> send it to 36474@debbugs.gnu.org.

> Please do not send mail to help-debbugs@gnu.org unless you wish
> to report a problem with the Bug-tracking system.

> -- 
> 36474: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=36474
> GNU Bug Tracking System
> Contact help-debbugs@gnu.org with problems

-- 
Alan Mackenzie (Nuremberg, Germany).