bug#5122: Mismatched parentheses when dealing with huge buffercontent

bug#5122: Mismatched parentheses when dealing with huge buffercontent

[Drew Adams]

See option `blink-matching-paren-distance' and node Matching in the Emacs
manual.

Does increasing that variable's value resolve the problem? 

bug#5122: Mismatched parentheses when dealing with huge buffercontent

[Deniz Dogan]

2009/12/4 Drew Adams <drew.adams@oracle.com>:
> See option `blink-matching-paren-distance' and node Matching in the Emacs
> manual.
>
> Does increasing that variable's value resolve the problem?
>
>

Yes, it does solve the problem.

-- 
Deniz Dogan

bug#5122: Mismatched parentheses when dealing with huge buffercontent

[Drew Adams]

> > See option `blink-matching-paren-distance' and node 
> > Matching in the Emacs manual.
> > Does increasing that variable's value resolve the problem?
> 
> Yes, it does solve the problem.

It's not real obvious when you see mismatch highlighting (and are used to it
working) that the highlighting might not mean a mismatch at all, but just an
insufficiently large option value. Many (most?) users will not even guess that
there is such an option.

Perhaps we could use a :help-echo property along with the mismatch highlight
face, pointing to the option?

bug#5122: Mismatched parentheses when dealing with huge buffercontent

[Deniz Dogan]

2009/12/5 Drew Adams <drew.adams@oracle.com>:
>> > See option `blink-matching-paren-distance' and node
>> > Matching in the Emacs manual.
>> > Does increasing that variable's value resolve the problem?
>>
>> Yes, it does solve the problem.
>
> It's not real obvious when you see mismatch highlighting (and are used to it
> working) that the highlighting might not mean a mismatch at all, but just an
> insufficiently large option value. Many (most?) users will not even guess that
> there is such an option.
>
> Perhaps we could use a :help-echo property along with the mismatch highlight
> face, pointing to the option?
>
>

I'm not sure what the best solution would be, but it is indeed strange
that Emacs says that there was a parenthesis *mismatch*. Maybe it
would be better to say something along the lines of "Emacs was unable
to find a matching parenthesis in the previous X characters", but
obviously much more terse.

bug#5122: Mismatched parentheses when dealing with huge buffercontent

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Fri, 4 Dec 2009 16:31:51 -0800
> Cc: emacs-pretest-bug@gnu.org, 5122@emacsbugs.donarmstrong.com
> 
> It's not real obvious when you see mismatch highlighting (and are used to it
> working) that the highlighting might not mean a mismatch at all, but just an
> insufficiently large option value. Many (most?) users will not even guess that
> there is such an option.
> 
> Perhaps we could use a :help-echo property along with the mismatch highlight
> face, pointing to the option?

help-echo is not guaranteed to be seen, and could be tricky when
there's an important message in the echo area (especially if the
help-echo itself is displayed in the echo area).  Also, it will not
work on a text terminal.

I think it's better to modify the echo area message, such that if
Emacs hits the blink-matching-paren-distance limit (as opposed to
hitting point-min), the message says something like

  Mismatched parentheses (within blink-matching-paren-distance)

bug#5122: Mismatched parentheses when dealing with huge buffercontent

[Drew Adams]

> > It's not real obvious when you see mismatch highlighting 
> (and are used to it
> > working) that the highlighting might not mean a mismatch at 
> all, but just an
> > insufficiently large option value. Many (most?) users will 
> not even guess that
> > there is such an option.
> > 
> > Perhaps we could use a :help-echo property along with the 
> mismatch highlight
> > face, pointing to the option?
> 
> help-echo is not guaranteed to be seen, and could be tricky when
> there's an important message in the echo area (especially if the
> help-echo itself is displayed in the echo area).  Also, it will not
> work on a text terminal.
> 
> I think it's better to modify the echo area message, such that if
> Emacs hits the blink-matching-paren-distance limit (as opposed to
> hitting point-min), the message says something like
> 
>   Mismatched parentheses (within blink-matching-paren-distance)

Yes that sounds good too.

To be honest, I don't know if I ever noticed any message in the echo area for
this. Perhaps because the message was so expected I never paid any attention to
it.

bug#5122: Mismatched parentheses when dealing with huge buffercontent

[Chong Yidong]

> I think it's better to modify the echo area message, such that if
> Emacs hits the blink-matching-paren-distance limit (as opposed to
> hitting point-min), the message says something like
>
>   Mismatched parentheses (within blink-matching-paren-distance)

Error messages shouldn't be so verbose---they can't spell out every bell
and whistle related to the error.  I prefer leaving things the way they
are.

bug#5122: Mismatched parentheses when dealing with huge buffercontent

[Stefan Monnier]

>> I think it's better to modify the echo area message, such that if
>> Emacs hits the blink-matching-paren-distance limit (as opposed to
>> hitting point-min), the message says something like
>> Mismatched parentheses (within blink-matching-paren-distance)
> Error messages shouldn't be so verbose---they can't spell out every bell
> and whistle related to the error.  I prefer leaving things the way they
> are.

But it should be less definitive: rather than just say authoritatively
"mismatched paren", it should at least be more humble and say "can't
find matching paren".


        Stefan

bug#5122: Mismatched parentheses when dealing with huge buffercontent

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: 5122@emacsbugs.donarmstrong.com
> Date: Sat, 05 Dec 2009 15:57:58 -0500
> 
> > I think it's better to modify the echo area message, such that if
> > Emacs hits the blink-matching-paren-distance limit (as opposed to
> > hitting point-min), the message says something like
> >
> >   Mismatched parentheses (within blink-matching-paren-distance)
> 
> Error messages shouldn't be so verbose---they can't spell out every bell
> and whistle related to the error.

I'd say adding a couple of words is hardly verbose, and certainly
isn't spelling out every detail.

> I prefer leaving things the way they are.

And leave users without a clue? how's that preferable?

bug#5122: Mismatched parentheses when dealing with huge buffercontent

[Chong Yidong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> But it should be less definitive: rather than just say authoritatively
> "mismatched paren", it should at least be more humble and say "can't
> find matching paren".

"No matching paren found" sounds fine.

bug#5122: Mismatched parentheses when dealing with huge buffercontent

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

>> Error messages shouldn't be so verbose---they can't spell out every bell
>> and whistle related to the error.
>
> I'd say adding a couple of words is hardly verbose, and certainly
> isn't spelling out every detail.
>
>> I prefer leaving things the way they are.
>
> And leave users without a clue? how's that preferable?

The circumstances in which the parenthesis are mismatched probably
vastly outweigh the circumstances in which the user wants to change
blink-matching-paren-distance.

So, it is arguably more confusing to the average user if the error
message mumbles something about blink-matching-paren-distance.  For
those who care, it's an apropos search away.

bug#5122: Mismatched parentheses when dealing with hugebuffercontent

[Drew Adams]

> >> Error messages shouldn't be so verbose---they can't spell 
> >> out every bell and whistle related to the error.
> >
> > I'd say adding a couple of words is hardly verbose, and certainly
> > isn't spelling out every detail.
> >
> >> I prefer leaving things the way they are.
> >
> > And leave users without a clue? how's that preferable?

Eli is right.

It is plain poor judgment to leave things the way they are in this case.
 
> The circumstances in which the parenthesis are mismatched probably
> vastly outweigh the circumstances in which the user wants to change
> blink-matching-paren-distance.

That's one point, sure. But the opposing consideration is how much damage occurs
when the problem, rare though it might be, does arise.

When a false mismatch occurs, the user has no clue - truly no clue - as to what
is going on. (This is in fact truer the rarer the event is!)

I know this because it happened to me. That's why I was able to respond to
Deniz's bug report so quickly. I just went through the same thing a month or so
back.

(And yes, I had already known this years ago, but had forgotten - perhaps
because of the rarity. I was looking at code from someone else (BBDB), which
uses much larger function definitions than I do.)

I spent quite a bit of time trying, in vain, to figure out what the problem was
with the code (e.g. an unescaped left paren in col 1 of a doc string, escape
sequences in regexp strings,...).

And that's the point, the opposing consideration: the difficulty and
improbability of finding the explanation/solution. One's reflex is not to
suppose that this is normal behavior governed by some user option. Believe me.
The tendency is to try to find out what is causing the mismatch. No one
spontaneously doubts that there is a mismatch.

This is very different from some other limits we have in Emacs. When a buffer is
bigger than the default font-locking buffer size, you can tell: there simply is
no font-locking. So if you're ignorant of the cause/remedy in that case, you
naturally search for something having to do with font-locking - you can tell
that there is a font-locking problem. You search a bit, and you eventually find
`font-lock-fontify-buffer' and solve the problem.

But here the natural inclination is to assume, since there is mismatch
highlighting, that there is, well, a mismatch. Duh.

I said the user has no clue as to what's going on in this situation. But it's
actually worse than that. We lead the user down the garden path. We say, in
effect, "Look over there! (without saying where)".

Gotcha! Look all you want over there, but there is nothing to find. The problem
is right here. It lies with the messenger who's telling you to look over there.

The paren-mismatch-highlighting code itself _knows_ when it has reached its
limit (or it should know), so it _knows_ when its highlighting no longer makes
any sense and can only mislead. It is up to that code to DTRT and - at a minimum
- do nothing in this context; do no harm. Better would be for it to signal
explicitly that it has gone beyond its limit.

> So, it is arguably more confusing to the average user if the error
> message mumbles something about blink-matching-paren-distance.  For
> those who care, it's an apropos search away.

You seem to have an interesting understanding of what is confusing to users,
average or otherwise.

And I couldn't disagree more about the ease of finding the problem, "for those
who care". To find it, you need to be looking for it. You need to have some idea
of what is going on, some idea that the paren highlighting is not functioning
normally and its helpful indicator should not be trusted. You need some hint
that you should look for an option concerning parens.

When you see mismatch highlighting, you do not assume that something is wrong
with the highlighting (or that some limit has been surpassed for its normal
functioning). You assume instead that there is in fact a mismatch somewhere.
_You trust what Emacs tells you._

Imagine if compiler error messages showed up spuriously, incorrectly, whenever
some esoteric threshold was surpassed - a setting that you didn't even know
about. Would you even think to look for such a setting? Or would you instead,
naturally, believe the compiler error message and dive into a close examination
of your code to try to understand its problem. Yes, you would look over there...

Emacs is misleading users in this situation. It is staring up at the top of a
building, causing others to look up also, even though there is nothing to see
there.

If you are really worried about the intrusiveness of an explanation in the echo
message or in a tooltip (remember, the mouse would need to be over the
"mismatched" paren for the tooltip message to appear, and the message would only
need to be displayed when the threshold is in fact surpassed), then consider
simply _not highlighting_ at all whenever the search distance has been exceeded.

Turning off highlighting in this case would at least clue the user to look for
something having to do with paren highlighting. What is bad design ("confusing
to the average user") is blindly signaling an error (mismatch) when the
measuring tool knows (or should know) that it is currently incapable of
measuring at all!

[Also, this option/feature should not necessarily be called "blink" anything, or
at least an alias should be provided, so that it would be easier to find for
those who use `show-paren-mode'. If one uses paren highlighting, then this is
more obviously about paren highlighting than paren blinking.]

bug#5122: Mismatched parentheses when dealing with huge buffercontent

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: 5122@emacsbugs.donarmstrong.com
> Date: Sat, 05 Dec 2009 17:11:09 -0500
> 
> The circumstances in which the parenthesis are mismatched probably
> vastly outweigh the circumstances in which the user wants to change
> blink-matching-paren-distance.

I suggested the modified message text _only_ for those cases where the
search hits the limit.

bug#5122: Mismatched parentheses when dealing with hugebuffercontent

[Drew Adams]

> If you are really worried about the intrusiveness of an 
> explanation in the echo message or in a tooltip (remember,
> the mouse would need to be over the "mismatched" paren
> for the tooltip message to appear, and the message would
> only need to be displayed when the threshold is in fact 
> surpassed), then consider simply _not highlighting_ at all
> whenever the search distance has been exceeded.

Another suggestion - when the limit is reached:

1. Don't highlight this "mismatch".

2. Display message stating that we hit the limit defined by the option.

3. Open the help buffer for the option.

#3 is the new part of the suggestion. It lets users get complete info about the
option, and lets them click to go to Customize to change the value (e.g.
temporarily).

Otherwise, they need to open *Messages*, get the variable name, and then use
`set-variable' or `M-x customize-option', to change the value.

bug#5122: Mismatched parentheses when dealing with huge buffercontent

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

>> The circumstances in which the parenthesis are mismatched probably
>> vastly outweigh the circumstances in which the user wants to change
>> blink-matching-paren-distance.
>
> I suggested the modified message text _only_ for those cases where the
> search hits the limit.

How do you distinguish this from the plain old "no mismatched
parenthesis" case?  Like I said, if there is no matching parenthesis
within the search limit (which, btw, has been increased in Emacs 22.3),
in most cases the parentheses really are mismatched.

bug#5122: Mismatched parentheses when dealing with hugebuffercontent

[Drew Adams]

> >> The circumstances in which the parenthesis are mismatched probably
> >> vastly outweigh the circumstances in which the user wants to change
> >> blink-matching-paren-distance.
> >
> > I suggested the modified message text _only_ for those 
> > cases where the search hits the limit.
> 
> How do you distinguish this from the plain old "no mismatched
> parenthesis" case?  Like I said, if there is no matching parenthesis
> within the search limit (which, btw, has been increased in 
> Emacs 22.3), in most cases the parentheses really are mismatched.

The code lets you distinguish. It simply needs to DTRT when it knows that the
arbitrary limit was reached.

Look at the code in both simple.el and paren.el. The code knows when it is using
the limit (narrowing etc.). Currently, it doesn't distinguish this case from the
case where it simply hits bob/eob, but it certainly could.

(No, I won't be sending a patch for this. If the will is there, you will see
easily enough what needs to be done.)

bug#5122: Mismatched parentheses when dealing with huge buffercontent

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: 5122@emacsbugs.donarmstrong.com
> Date: Sun, 06 Dec 2009 10:30:37 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> The circumstances in which the parenthesis are mismatched probably
> >> vastly outweigh the circumstances in which the user wants to change
> >> blink-matching-paren-distance.
> >
> > I suggested the modified message text _only_ for those cases where the
> > search hits the limit.
> 
> How do you distinguish this from the plain old "no mismatched
> parenthesis" case?

Sorry, I don't understand the question.  What is the "no mismatched
parenthesis" case?

To clarify, my suggestion was to display

  Mismatched parenthesis

if the search hits point-min or point-max without finding a match, or

  Mismatched parenthesis (within blink-matching-paren-distance)

if the search doesn't find a match within
blink-matching-paren-distance.

> Like I said, if there is no matching parenthesis within the search
> limit (which, btw, has been increased in Emacs 22.3), in most cases
> the parentheses really are mismatched.

But we don't know that, unless we searched all the way to the
beginning or end of the buffer.  All we know is that we didn't find a
match within the distance searched.

bug#5122: Mismatched parentheses when dealing with huge buffercontent

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

>> Like I said, if there is no matching parenthesis within the search
>> limit (which, btw, has been increased in Emacs 22.3), in most cases
>> the parentheses really are mismatched.
>
> But we don't know that, unless we searched all the way to the
> beginning or end of the buffer.  All we know is that we didn't find a
> match within the distance searched.

I understand what you're saying.  But even if there is a "matching"
paren more than 102k characters away (the blink-matching-paren-distance
default), chances are that's a mistake anyway.

Yes, I know the failure condition exists.  But for every user
enlightened by a more verbose error message, I think there will be
hundreds more that will be unnecessarily confused.

bug#5122: Mismatched parentheses when dealing with hugebuffercontent

[Chong Yidong]

"Drew Adams" <drew.adams@oracle.com> writes:

> Eli is right.
>
> It is plain poor judgment to leave things the way they are in this case.

If you can't learn to disagree without being disagreeable, few will have
the incentive to wade through your "War And Peace By Leo Tolstoy"-style
emails (certainly not me).

bug#5122: Mismatched parentheses when dealing with hugebuffercontent

[Drew Adams]

> >> Like I said, if there is no matching parenthesis within the search
> >> limit (which, btw, has been increased in Emacs 22.3), in most cases
> >> the parentheses really are mismatched.
> >
> > But we don't know that, unless we searched all the way to the
> > beginning or end of the buffer.  All we know is that we 
> > didn't find a match within the distance searched.
> 
> I understand what you're saying.  But even if there is a "matching"
> paren more than 102k characters away (the 
> blink-matching-paren-distance default), chances are that's a
> mistake anyway.

Why would that be true any more than if the limit were 300 chars instead of
102k? The matching code works to the same degree, no matter what the limit is.
Either the matching code is accurate, or it is not, but the limit used has no
bearing on that.

The cases where what we are asking for is needed are real cases where large
function definitions (or other large lists in code) exist. The code is not a
"mistake", even if it might be poor style. What is a mistake is to report a
paren mismatch when there is no mismatch.

No one has yet pointed to any case where the matching code has shown a false
positive, claiming a mismatch where there isn't any - except when the search is
limited, which is precisely the case in question. We can and do rely on the
accuracy of mismatch reporting, when the search is not prematurely ended by the
limit. It works well. Do you have some reason to doubt it?

> Yes, I know the failure condition exists.  But for every user
> enlightened by a more verbose error message, I think there will be
> hundreds more that will be unnecessarily confused.

Why would anyone be confused by a message saying that the search limit was
reached? 

Turning off the mismatch highlighting in this case, and displaying the help
buffer for the limit option (with its link to Customize), would dispel any
possible confusion about what is happening.

Massive confusion arises currently, though, by incorrectly showing mismatch
highlighting when there is no mismatch. That completely disorienting confusion
occurs with _exactly_ the same frequency as would the message that (only) you
think would be confusing. This situation is rare. But it is important (yes, for
the average user) to help, not hurt, when it does arise.

bug#5122: Mismatched parentheses when dealing with hugebuffercontent

[Drew Adams]

> > Eli is right.
> >
> > It is plain poor judgment to leave things the way they are 
> > in this case.
> 
> If you can't learn to disagree without being disagreeable, 
> few will have the incentive to wade through your "War And
> Peace By Leo Tolstoy"-style emails (certainly not me).

I'm sorry you feel that way.

But there is nothing in the statement I made that is disagreeable or personal -
certainly nothing like that was intended. We are making a judgment (design
decision). Leaving things the way they are now would, IMHO, be a poor choice.

The proposal to do that was from you, yes, but that's not my fault. It is your
proposal I oppose, not you. If you insist on taking my statement personally I
cannot stop you, but that is not productive and that was not my intention. I'm
sorry if it came across that way to you.

I think the _idea_ you promoted is a poor one - and I gave concrete reasons why.
No one is immune from making bad proposals (certainly not me). I did not say
anything about _you_; it is the design you defended that I opposed, and in
specific terms.

I am not responsible for _who_ might choose to defend the status quo design
here. My argument is that we should change it. I would make exactly the same
arguments against it, no matter who defended it. I'm not interested in attacking
anyone; I'm interested in an exchange of ideas that improves Emacs for users.

So don't take it personally - try examining the _arguments_. You will find them,
as well as concrete solution suggestions, in the emails you say you don't want
to read.

Instead of singling out an intro statement where I said that keeping the status
quo is a bad idea, try moving on to read what I said about _why_ and what to do
about it. Put that statement in context.

My mail said essentially "this is a bad idea because...", and all you chose to
read, apparently, was "this is a bad idea". Was it because it was your idea that
you didn't care to read past that? If so, that's not a very good reason.

I've written several mails now in this thread. Judge their technical content -
and their tone - as a whole. I stand by what I wrote - feel free to counter any
part of it technically - that's how we advance. Forget about who wrote what, and
just address the ideas themselves.

I'm willing to address your arguments. You ignore mine, ostensibly because of my
writing style. That sounds like a convenient excuse not to confront the
arguments. It seems it is you who is attacking the messenger, not I.

I believe you mischaracterize my contribution here, painting it as essentially
unproductive ranting to be ignored. That might be convenient if you don't have
counter arguments, but it is not accurate, or fair, or productive.

bug#5122: Mismatched parentheses when dealing with huge buffercontent

[Stefan Monnier]

>> I prefer leaving things the way they are.
> And leave users without a clue? how's that preferable?

Clearly, the only real answer is to get rid of
blink-matching-paren-distance.  We could try and do that, maybe using
while-no-input.


        Stefan

bug#5122: Mismatched parentheses when dealing with huge buffercontent

[MON KEY]

For really large groups of parens of type `mismatch-maybe' would it be
possible/reasonable/faster to do a check/balance to the nearest pair
of parens relative
the first 10 Catalan numbers e.g. something like this (or his brethren):

(URL `http://www2.research.att.com/~njas/sequences/A000108')
(URL `http://www2.research.att.com/~njas/sequences/table?a=108&fmt=4')

--
s_P

bug#5122: Mismatched parentheses when dealing with huge buffercontent

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> I prefer leaving things the way they are.
>> And leave users without a clue? how's that preferable?
>
> Clearly, the only real answer is to get rid of
> blink-matching-paren-distance.  We could try and do that, maybe using
> while-no-input.

That would be a good long-term solution.

Meanwhile, I've implemented Eli's suggestion of just altering the
message to add " within the limit" if we're limited.  So I'm closing
this bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#5122: Mismatched parentheses when dealing with huge buffercontent

[Drew Adams]

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> >>> I prefer leaving things the way they are.
> >>
> >> And leave users without a clue? how's that preferable?
> >
> > Clearly, the only real answer is to get rid of
> > blink-matching-paren-distance.  We could try and do that, 
> > maybe using while-no-input.
> 
> That would be a good long-term solution.
> 
> Meanwhile, I've implemented Eli's suggestion of just altering the
> message to add " within the limit" if we're limited.  So I'm closing
> this bug report.

Stefan says what "clearly, the only real answer" is.
He says that can be tried, and he makes a suggestion how.

So why, if that is clearly the answer, do you close the bug?

Makes no sense.  Is the goal just to close bugs or to fix the problem?

Is someone getting paid by the bug closure or something? ;-)

bug#5122: Mismatched parentheses when dealing with huge buffercontent

[Drew Adams]

> Stefan says what "clearly, the only real answer" is.
> He says that can be tried, and he makes a suggestion how.
> 
> So why, if that is clearly the answer, do you close the bug?
> Makes no sense.  Is the goal just to close bugs or to fix the problem?
> Is someone getting paid by the bug closure or something? ;-)

And why close it as `fixed' if Emacs Dev will not fix it?
The problem reported is still there, AFAIK.  Why not `wontfix'?