Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Alan Mackenzie]

Hello, Emacs.

Bug #22983 is that syntax-ppss is broken.  It's text is this:

    The fundamental contract in syntax-ppss is that (syntax-ppss POS)
    returns the same value as (parse-partial-sexp (point-min) POS) (with the
    exception of elements 2 and 6).  This is currently not always the case.

    In the master branch, emacs -Q and visit xdisp.c with C-x C-f.  Follow
    this recipe:

        M-: (syntax-ppss-flush-cache 1)
        M-: (setq ppss-0 (syntax-ppss 40000))
        M-<
        C-s #include " <CR>
        M->
        C-x n n
        M-: (setq ppss-1 (syntax-ppss 40000))
        M-: (setq parse (parse-partial-sexp (point-min) 40000))

    At this point, `ppss-1' and `parse' should match (apart from elements 2
    and 6).  What we actually have is:

        ppss-1: (2 39992 nil nil nil nil 2 nil nil (39975 39992))
        parse:  (0 nil 15674 34 nil nil 0 nil 15675 nil)

    . 

.  I've just tried it again in the emacs-25 branch, and it is as badly
broken as it was back in March.

Personally, I believe that syntax-ppss is such a bad function, we should
work to expunge it from Emacs.  However, I recognise that that is a view
unlikely to prevail.

My suggestion for a fix, discussed in the thread for the bug, is to come
up with a new function similar to syntax-ppss which always returns the
equivalent of

    (parse-partial-sexp 1 pos)

in place of syntax-ppss's

    (parse-partial-sexp (point-min) pos)

.  Being realistic, this new function would have to bear the name
"syntax-ppss", leaving lots of room for confusion, but it's probably the
best fix we'll get.  I'm willing to make and commit this change.

With this fix, and using this new function in place of the current
syntax-ppss in accordance with its spec, the above bug wouldn't happen.

If my fix isn't wanted, can somebody who likes syntax-ppss PLEASE fix it
before the release, so that, finally, it behaves according to its
specification.

Thanks!

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Dmitry Gutov]

On 06/08/2016 01:09 AM, Alan Mackenzie wrote:

> I've just tried it again in the emacs-25 branch, and it is as badly
> broken as it was back in March.

I think it was considered to be too risky a change even back in March, 
and it was never discussed as a possible blocker.

But that's not for me to decide.

> With this fix, and using this new function in place of the current
> syntax-ppss in accordance with its spec, the above bug wouldn't happen.
>
> If my fix isn't wanted, can somebody who likes syntax-ppss PLEASE fix it
> before the release, so that, finally, it behaves according to its
> specification.

Is there something wrong with the patch I posted?

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22983#47

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Alan Mackenzie]

Hello, Dmitry.

On Wed, Jun 08, 2016 at 01:15:47AM +0300, Dmitry Gutov wrote:
> On 06/08/2016 01:09 AM, Alan Mackenzie wrote:

> > I've just tried it again in the emacs-25 branch, and it is as badly
> > broken as it was back in March.

> I think it was considered to be too risky a change even back in March, 
> and it was never discussed as a possible blocker.

I think I'd be happy enough about it being fixed in master, if the fix
happens before the release.  :-).  The trouble is, if there's no
deadline, it'll never get fixed.

> But that's not for me to decide.

> > With this fix, and using this new function in place of the current
> > syntax-ppss in accordance with its spec, the above bug wouldn't happen.

> > If my fix isn't wanted, can somebody who likes syntax-ppss PLEASE fix it
> > before the release, so that, finally, it behaves according to its
> > specification.

> Is there something wrong with the patch I posted?

The one on 2016-03-10 02:49:34 +0200?

Yes, I think having the binary toggle `syntax-ppss-dont-widen' purely to
direct the innards of the function is poor programming (since it
explicitly toggles a toggle inside a supposedly abstract function).  I
think an improvement would be to dispense with that toggle, and have two
distinct functions, one in place of `syntax-ppss-dont-widen' being nil,
and the other in place of `s-p-d-w' being non-nil.  The latter function
might usefully have an extra parameter specifying the base point that
parse-partial-sexp should be calculated from.  That would leave quite a
few options open for the internal logic of the function.

Or something like that.

> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22983#47

Yes, that one.  I had trouble finding it on the web archive.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Dmitry Gutov]

On 06/08/2016 01:48 AM, Alan Mackenzie wrote:

> I think I'd be happy enough about it being fixed in master, if the fix
> happens before the release.  :-).  The trouble is, if there's no
> deadline, it'll never get fixed.

Why before the release in particular?

If you're interested in having it fixed sooner, please look into helping 
Vitalie in 
http://lists.gnu.org/archive/html/emacs-devel/2016-04/msg00850.html. 
With hard-widen, maybe we won't need syntax-ppss-dont-widen after all.

> Yes, I think having the binary toggle `syntax-ppss-dont-widen' purely to
> direct the innards of the function is poor programming (since it
> explicitly toggles a toggle inside a supposedly abstract function).

Abstract?

> I
> think an improvement would be to dispense with that toggle, and have two
> distinct functions, one in place of `syntax-ppss-dont-widen' being nil,
> and the other in place of `s-p-d-w' being non-nil.  The latter function
> might usefully have an extra parameter specifying the base point that
> parse-partial-sexp should be calculated from.  That would leave quite a
> few options open for the internal logic of the function.

That wouldn't help in the multi-mode case, which is the primary use I 
have in mind for syntax-ppss-dont-widen.

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Alan Mackenzie]

Hello, Dmitry.

On Wed, Jun 08, 2016 at 02:25:22AM +0300, Dmitry Gutov wrote:
> On 06/08/2016 01:48 AM, Alan Mackenzie wrote:

> > I think I'd be happy enough about it being fixed in master, if the fix
> > happens before the release.  :-).  The trouble is, if there's no
> > deadline, it'll never get fixed.

> Why before the release in particular?

Haven't I just said?  :-)  If not before the release, then when?

> If you're interested in having it fixed sooner, please look into helping 
> Vitalie in 
> http://lists.gnu.org/archive/html/emacs-devel/2016-04/msg00850.html. 
> With hard-widen, maybe we won't need syntax-ppss-dont-widen after all.

Is Vitalie still working on this (or indeed, anything else to do with
Emacs)?  I don't recall seeing him for quite a number of weeks, now.

I can't say I feel enthusiastic about "hard-widen".  I haven't looked
into it that thoroughly, but it feels a bit like an ugly hack whose
ramifications might percolate all the way through Emacs, and that might
make us regret it fairly heavily in the not too distant future.

> > Yes, I think having the binary toggle `syntax-ppss-dont-widen' purely to
> > direct the innards of the function is poor programming (since it
> > explicitly toggles a toggle inside a supposedly abstract function).

> Abstract?

Yes, abstract.  `syntax-ppss' has an specification and interface on the
one side, and an implementation on the other.  This separation is an
abstraction.  Although it might not be a very strong one, given how
users must keep "servicing" it, it is better than nothing.

> > I think an improvement would be to dispense with that toggle, and
> > have two distinct functions, one in place of
> > `syntax-ppss-dont-widen' being nil, and the other in place of
> > `s-p-d-w' being non-nil.  The latter function might usefully have an
> > extra parameter specifying the base point that parse-partial-sexp
> > should be calculated from.  That would leave quite a few options
> > open for the internal logic of the function.

> That wouldn't help in the multi-mode case, which is the primary use I 
> have in mind for syntax-ppss-dont-widen.

Why would it not help?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Dmitry Gutov]

On 06/11/2016 01:07 PM, Alan Mackenzie wrote:

>> Why before the release in particular?
>
> Haven't I just said?  :-)  If not before the release, then when?

Right after? Slightly after that? Whenever. "Before the release" doesn't 
look like a meaningful constraint, considering the commit will go on master.

I'm still not even clear which branch 25.2 will come from, and that's an 
important concern.

> Is Vitalie still working on ... anything else to do with
> Emacs)?

Yes:

https://github.com/emacs-ess/ESS/commits/master
https://github.com/vspinu/polymode/commits/master

Adding hard-widen, however, is a low-level endeavor, and it seems nobody 
sufficiently proficient there is in any hurry to help.

> I can't say I feel enthusiastic about "hard-widen".  I haven't looked
> into it that thoroughly, but it feels a bit like an ugly hack whose
> ramifications might percolate all the way through Emacs, and that might
> make us regret it fairly heavily in the not too distant future.

Hadn't you yourself proposed a far more invasive solution, not too long 
ago? I think hard-widen strikes a certain balance, in that is will 
require a moderate amount of changes in Emacs, and little to no changes 
in the user code, considering we've always (or at least for a long time) 
had to write it with consideration for the current narrowing.

If there's anything to complain about it, it's that it doesn't bring new 
niceties. So: modest gains, and little amount of pain. Seems 
conservative enough for the Emacs community.

>>> Yes, I think having the binary toggle `syntax-ppss-dont-widen' purely to
>>> direct the innards of the function is poor programming (since it
>>> explicitly toggles a toggle inside a supposedly abstract function).
>
>> Abstract?
>
> Yes, abstract.  `syntax-ppss' has an specification and interface on the
> one side, and an implementation on the other.

Just like any other function?

>>> I think an improvement would be to dispense with that toggle, and
>>> have two distinct functions, one in place of
>>> `syntax-ppss-dont-widen' being nil, and the other in place of
>>> `s-p-d-w' being non-nil.  The latter function might usefully have an
>>> extra parameter specifying the base point that parse-partial-sexp
>>> should be calculated from.  That would leave quite a few options
>>> open for the internal logic of the function.
>
>> That wouldn't help in the multi-mode case, which is the primary use I
>> have in mind for syntax-ppss-dont-widen.
>
> Why would it not help?

Because you (in CC Mode) will choose the more straightforward version 
that does widen. And I, in mmm-mode, will thus have no ability to modify 
what your code is doing by narrowing, and will send you dark thoughts 
instead.

What's the purpose of the other function? Who's going to use it?

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Alan Mackenzie]

Hello, Dmitry.

On Sat, Jun 11, 2016 at 07:10:03PM +0300, Dmitry Gutov wrote:
> On 06/11/2016 01:07 PM, Alan Mackenzie wrote:

> I'm still not even clear which branch 25.2 will come from, and that's an 
> important concern.

My guess is it'll will come from the master branch.  Otherwise, there'll
be too much work in cherry picking commits back to the emacs-25 branch.
That's just a guess, though.

> > Is Vitalie still working on ... anything else to do with
> > Emacs)?

> Yes:

> https://github.com/emacs-ess/ESS/commits/master
> https://github.com/vspinu/polymode/commits/master

Glad to hear that!

> Adding hard-widen, however, is a low-level endeavor, and it seems nobody 
> sufficiently proficient there is in any hurry to help.

> > I can't say I feel enthusiastic about "hard-widen".  I haven't looked
> > into it that thoroughly, but it feels a bit like an ugly hack whose
> > ramifications might percolate all the way through Emacs, and that might
> > make us regret it fairly heavily in the not too distant future.

> Hadn't you yourself proposed a far more invasive solution, not too long 
> ago?

Intrusive it is, but an ugly hack it certainly isn't.  It's aim is to
bring Emacs to the state it would have been in if it had been designed
for multiple major modes per buffer right from the start.  As such it
will be free from ugly workarounds.

The main problem with it is the amount of work to implement it is huge.
So much so, that I might well never finish it myself.  There was not a
great deal of enthusiasm expressed for it all these weeks ago, apart from
from Eli and yourself, and even then, Eli was unhappy about a fairly
important part of it.

> I think hard-widen strikes a certain balance, in that is will require a
> moderate amount of changes in Emacs, and little to no changes in the
> user code, considering we've always (or at least for a long time) had
> to write it with consideration for the current narrowing.

hard-widen seems to destroy the simplicity evident in narrow-to-region
and widen.  It's an ugly hack, and I fear that if we implement it, it
will have ugly unforeseen consequences which it will then be too late to
do anything about.

Why does (presumably) a super mode want to do anything with narrowing
anyway?  Narrowing is needed by users and by major modes, and its use by
a super mode might well clash.  Is this hard-widen proposal at all
documented in any coherent fashion?

> If there's anything to complain about it, it's that it doesn't bring new 
> niceties. So: modest gains, and little amount of pain. Seems 
> conservative enough for the Emacs community.

I'm not convinced.  But then, I don't know how the proposal will work in
any great detail.

> >>> Yes, I think having the binary toggle `syntax-ppss-dont-widen' purely to
> >>> direct the innards of the function is poor programming (since it
> >>> explicitly toggles a toggle inside a supposedly abstract function).

> >> Abstract?

> > Yes, abstract.  `syntax-ppss' has an specification and interface on the
> > one side, and an implementation on the other.

> Just like any other function?

No, some so-called functions, though thankfully few in Emacs, are
incoherent assemblages, with callers having to reach in and manipulate the
innards (and hence be aware of the implementation).  And so on.  Look up
terms like "coherence" and "coupling" in computer science text books.

> >>> I think an improvement would be to dispense with that toggle, and
> >>> have two distinct functions, one in place of
> >>> `syntax-ppss-dont-widen' being nil, and the other in place of
> >>> `s-p-d-w' being non-nil.  The latter function might usefully have an
> >>> extra parameter specifying the base point that parse-partial-sexp
> >>> should be calculated from.  That would leave quite a few options
> >>> open for the internal logic of the function.

> >> That wouldn't help in the multi-mode case, which is the primary use I
> >> have in mind for syntax-ppss-dont-widen.

> > Why would it not help?

> Because you (in CC Mode) will choose the more straightforward version 
> that does widen. And I, in mmm-mode, will thus have no ability to modify 
> what your code is doing by narrowing, and will send you dark thoughts 
> instead.

That paragraph worries me greatly.  OF COURSE CC Mode uses
narrow-to-region and widen freely, there being code which can scarcely
work without it.  I and other major mode maintainers will expect to
continue to be able to do so.  A super mode should, should it use
widening and narrowing, do so in a manner which won't interfere with
major modes' and users' use of it.

> What's the purpose of the other function? Who's going to use it?

Maybe nobody.  ;-)  What I missed was the need to insert it far and wide
into existing code.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Dmitry Gutov]

On 06/11/2016 10:50 PM, Alan Mackenzie wrote:

>> I'm still not even clear which branch 25.2 will come from, and that's an
>> important concern.
>
> My guess is it'll will come from the master branch.  Otherwise, there'll
> be too much work in cherry picking commits back to the emacs-25 branch.
> That's just a guess, though.

That sounds like we shouldn't commmit any significant changes to master, 
doesn't it?

>> Hadn't you yourself proposed a far more invasive solution, not too long
>> ago?
>
> Intrusive it is, but an ugly hack it certainly isn't.  It's aim is to
> bring Emacs to the state it would have been in if it had been designed
> for multiple major modes per buffer right from the start.  As such it
> will be free from ugly workarounds.

An ugly workaround is in the eye of a beholder. And it also depends on 
the context.

I believe hard-widen can be made to work well, and not too hard to 
reason about. Although only time could tell.

> The main problem with it is the amount of work to implement it is huge.
> So much so, that I might well never finish it myself.  There was not a
> great deal of enthusiasm expressed for it all these weeks ago, apart from
> from Eli and yourself, and even then, Eli was unhappy about a fairly
> important part of it.

Yes, the idea was big/complex, and the design is unprecedented among 
text editors, AFAIK. These two characteristics, taken together, make it 
it very risky, at least.

> hard-widen seems to destroy the simplicity evident in narrow-to-region
> and widen.

What simplicity? narrow-to-region seemingly can't decide whether it's a 
command for users to invoke, or it's something for Lisp code to use. We 
end up treating both cases the same, and that causes a lot of problems. 
hard-widen would differentiate those.

> Why does (presumably) a super mode want to do anything with narrowing
> anyway?

How else would you apply each major mode's fontifications only within 
its subregions?

See 
https://github.com/purcell/mmm-mode/blob/c9a857a638701482931ffaaee262b61ce53489f3/mmm-region.el#L789-L816

> Narrowing is needed by users and by major modes, and its use by
> a super mode might well clash.

super modes are currently implemented as minor modes. major modes 
shouldn't override the choices made by minor modes, if at all possible.

> Is this hard-widen proposal at all
> documented in any coherent fashion?

These have been several discussions.

> I'm not convinced.  But then, I don't know how the proposal will work in
> any great detail.

I've linked to the patch and the previous discussion not too long ago.

> That paragraph worries me greatly.  OF COURSE CC Mode uses
> narrow-to-region and widen freely, there being code which can scarcely
> work without it.  I and other major mode maintainers will expect to
> continue to be able to do so.

The main thing CC Mode would have to worry about from then on, is that 
it won't always be able to goto-char to positions beyond the hard 
narrowing, even if they exist in the buffer (and are pointed to by some 
buffer-local data structure maintained by CC Mode's parser).

There may be some alternative approaches to this, but a super mode has 
to be able to *somehow* limit CC Mode's activity if we want to be able 
to use it for syntax highlighting and indentation inside e.g. Noweb's 
code chunks.

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Alan Mackenzie]

Hello, Dmitry.

On Sun, Jun 12, 2016 at 01:52:42AM +0300, Dmitry Gutov wrote:
> On 06/11/2016 10:50 PM, Alan Mackenzie wrote:

[ .... ]

> I believe hard-widen can be made to work well, and not too hard to 
> reason about.

I can't reason about hard-widen very much, because I haven't read its
specification.

> Although only time could tell. 

No.  A careful consideration of its proposed working before
implementation (let's call it a design ;-) can reveal a lot about how
well it could work.

[ .... ]

> > hard-widen seems to destroy the simplicity evident in narrow-to-region
> > and widen.

> What simplicity?

`widen' makes the entire buffer accessible.  `narrow-to-region' makes
the specified region the accessible portion.  And that's it!  What could
be simpler?

That simplicity is a masterstroke in the early design of Emacs.

> narrow-to-region seemingly can't decide whether it's a command for
> users to invoke, or it's something for Lisp code to use.

That's the use case, not narrow-to-region itself.  It's actually both a
user command and a Lisp function, deliberately so, masterfully so.

> We end up treating both cases the same, and that causes a lot of
> problems.

I have encountered no such problems in over 15 years of hacking Emacs.
I think you're seeing problems somewhere and blaming
narrow-to-region/widen, when the real problem is somewhere else.

> hard-widen would differentiate those.

And that would cause great problems.  It would likely make
narrow-to-region/widen a pain to work with.

> > Why does (presumably) a super mode want to do anything with narrowing
> > anyway?

> How else would you apply each major mode's fontifications only within 
> its subregions?

By adding the appropriate structures as buffer local variables (or
perhaps as text properties) which would delimit the subregions, then
enhance Font Lock to respect those boundaries.

> See 
> https://github.com/purcell/mmm-mode/blob/c9a857a638701482931ffaaee262b61ce53489f3/mmm-region.el#L789-L816

That's rather a lot to take in before breakfast!

> > Narrowing is needed by users and by major modes, and its use by
> > a super mode might well clash.

> super modes are currently implemented as minor modes. major modes 
> shouldn't override the choices made by minor modes, if at all possible.

What????  What major modes shouldn't override is user settings, or
possibly minor mode settings.  The use/non-use of narrowing is NOT a
setting, any more than the use/non-use of cdr is.  They're just general
purpose tools.

> > Is this hard-widen proposal at all
> > documented in any coherent fashion?

> There have been several discussions.

They're not coherent, and they're not very useful.

Could you please write this document.  Given the massive change you want
to make in a fundamental Emacs tool, this is not unreasonable to ask.
This is, after all, what I did for the "islands" concept a couple of
months ago.  Your document need only be 50 - 100 lines long at the most.
Writing such doc would certainly help you clear things up for yourself,
too.

> > I'm not convinced.  But then, I don't know how the proposal will work in
> > any great detail.

> I've linked to the patch and the previous discussion not too long ago.

They are not useful for this purpose.

> > That paragraph worries me greatly.  OF COURSE CC Mode uses
> > narrow-to-region and widen freely, there being code which can scarcely
> > work without it.  I and other major mode maintainers will expect to
> > continue to be able to do so.

> The main thing CC Mode would have to worry about from then on, is that 
> it won't always be able to goto-char to positions beyond the hard 
> narrowing, even if they exist in the buffer (and are pointed to by some 
> buffer-local data structure maintained by CC Mode's parser).

That sounds like something worthwhile sorting out in advance.

> There may be some alternative approaches to this, but a super mode has 
> to be able to *somehow* limit CC Mode's activity if we want to be able 
> to use it for syntax highlighting and indentation inside e.g. Noweb's 
> code chunks.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Dmitry Gutov]

On 06/12/2016 12:34 PM, Alan Mackenzie wrote:

> I can't reason about hard-widen very much, because I haven't read its
> specification.

The change to the narrow-to-region's docstring in the patch contains the 
specification. Here's a better link to the message:

http://lists.gnu.org/archive/html/emacs-devel/2016-03/msg01576.html

> No.  A careful consideration of its proposed working before
> implementation (let's call it a design ;-) can reveal a lot about how
> well it could work.

...and time, if we're being realistic.

> `widen' makes the entire buffer accessible.  `narrow-to-region' makes
> the specified region the accessible portion.  And that's it!  What could
> be simpler?

If you take a shovel and remove its blade, the result will look very 
simple. That doesn't mean it's appropriate for any interesting task.

Nowhere does narrow-to-region's documentation say it should only be used 
in major modes.

> I have encountered no such problems in over 15 years of hacking Emacs.
> I think you're seeing problems somewhere and blaming
> narrow-to-region/widen, when the real problem is somewhere else.

It's been a known problem for a while, and it has come up in multiple 
discussions over the last years.

The "real problem" is a matter of perspective. Some might argue that 
trying to use narrowing at all, for any purpose, is a bad idea, and 
then, of course, the right choice would be to use something else.

>> How else would you apply each major mode's fontifications only within
>> its subregions?
>
> By adding the appropriate structures as buffer local variables (or
> perhaps as text properties) which would delimit the subregions, then
> enhance Font Lock to respect those boundaries.

Please go ahead and write up this proposal in detail. It doesn't seem as 
simple as you make it sound. And I wonder if it ends up to be different 
from the "islands" proposal.

Further, font-lock is not the only facility we want here. Making 
syntax-ppss obey region boundaries is another. There's also indentation 
code, which is more difficult to handle, but limiting visibility there 
somehow also seems essential.

The super mode can also adapt different simpler facilities, like Imenu, 
using the same approach (go over the chunks, narrow to each, scan using 
the language-appropriate tools, and concatenate the results).

>> See
>> https://github.com/purcell/mmm-mode/blob/c9a857a638701482931ffaaee262b61ce53489f3/mmm-region.el#L789-L816
>
> That's rather a lot to take in before breakfast!

Did you have the breakfast yet? It's just 30 lines.

>> super modes are currently implemented as minor modes. major modes
>> shouldn't override the choices made by minor modes, if at all possible.
>
> What????

What I said. To the extent the facilities allow, of course.

> The use/non-use of narrowing is NOT a
> setting, any more than the use/non-use of cdr is.  They're just general
> purpose tools.

Turns out, they are not general purpose enough. That's the problem.

>> There have been several discussions.
>
> They're not coherent, and they're not very useful.

If there's anything in them that makes you confused, I've yet to hear 
about it. As far as I can see, you're not interested in making even that 
effort in learning the subject.

> Could you please write this document.  Given the massive change you want
> to make in a fundamental Emacs tool, this is not unreasonable to ask.

I don't enjoy rehashing others' arguments when the original messages are 
readable enough.

It's a small change, by itself. Please read the discussion, starting 
with http://lists.gnu.org/archive/html/emacs-devel/2016-03/msg01129.html

And http://lists.gnu.org/archive/html/emacs-devel/2016-03/msg01182.html 
in particular. After that, please let me know if you have any specific 
questions.

> This is, after all, what I did for the "islands" concept a couple of
> months ago.

And I have spent considerable time contributing to that discussion as well.

>> The main thing CC Mode would have to worry about from then on, is that
>> it won't always be able to goto-char to positions beyond the hard
>> narrowing, even if they exist in the buffer (and are pointed to by some
>> buffer-local data structure maintained by CC Mode's parser).
>
> That sounds like something worthwhile sorting out in advance.

Sure. Please go ahead and present your thoughts on the subject.

I realize it might be rather complicated, actually, to use the narrowing 
approach with CC Mode as submode. But I don't see any alternative 
solutions that we're likely to implement. So it may be that some modes 
will still refuse to work in multi-mode context.

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 1401 bytes --]

On 2016-06-12 18:58, Dmitry Gutov wrote:
>> I have encountered no such problems in over 15 years of hacking
>> Emacs. I think you're seeing problems somewhere and blaming 
>> narrow-to-region/widen, when the real problem is somewhere else.
> 
> It's been a known problem for a while, and it has come up in multiple
> discussions over the last years.
> 
> The "real problem" is a matter of perspective. Some might argue that
> trying to use narrowing at all, for any purpose, is a bad idea, and
> then, of course, the right choice would be to use something else.

I can share one instance of running into issues with narrow-to-region. I was implementing a simple ‘presenter’ mode, such that special comments were used to delimit ‘slides’. The mode functioned by narrowing the buffer to regions delimited by these special comments. It was a very convenient way to page through code in front of an audience.

Unfortunately, this approach was not robust. I ran into issues with modes using (point-min) and (point-max) without widening, and thereby misplacing overlays. In the end things kept breaking, and I ended up implementing the same feature using two overlays with an invisible property, hiding everything before (respectively after) the region being shown. It was a pain.

Not sure whether this is a useful contribution :) In any case, that's one problem I ran into.

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Alan Mackenzie]

Hello, Dmitry.

On Mon, Jun 13, 2016 at 01:58:02AM +0300, Dmitry Gutov wrote:
> On 06/12/2016 12:34 PM, Alan Mackenzie wrote:

> > I can't reason about hard-widen very much, because I haven't read its
> > specification.

> The change to the narrow-to-region's docstring in the patch contains the 
> specification.

It does not.  It contains a few relatively vague aspects, undefined
terms, and "for full details, search!".  I'm not trying to slag off
Vitalie here, because clearly he was in the middle of trying to figure
things out, and was unclear about things.

> Here's a better link to the message:

> http://lists.gnu.org/archive/html/emacs-devel/2016-03/msg01576.html

I've glanced through that, and I found the doc string you've referred
to.  It's some help.

> > No.  A careful consideration of its proposed working before
> > implementation (let's call it a design ;-) can reveal a lot about how
> > well it could work.

> ...and time, if we're being realistic.

> > `widen' makes the entire buffer accessible.  `narrow-to-region' makes
> > the specified region the accessible portion.  And that's it!  What could
> > be simpler?

> If you take a shovel and remove its blade, the result will look very 
> simple. That doesn't mean it's appropriate for any interesting task.

So, you consider widen and narrow-to-region in their current form as
being as useless as a detached shovel blade?  I think you're just being
perverse, here.  widen and narrow-to-region are enormously useful, and
it is their simplicity, in large part, which makes them so useful.  You
are proposing to destroy this simplicity.

> Nowhere does narrow-to-region's documentation say it should only be used 
> in major modes.

Of course not.  As I've said, it's a general purpose tool used
univerally in Emacs.  I would like to keep it that way.

> > I have encountered no such problems in over 15 years of hacking Emacs.
> > I think you're seeing problems somewhere and blaming
> > narrow-to-region/widen, when the real problem is somewhere else.

> It's been a known problem for a while, and it has come up in multiple 
> discussions over the last years.

"It's been known"?  To whom?  The only discussion I have seen on the
topic has been between Stefan, yourself, and Vitalie.  Apologies if I've
missed anything.

> The "real problem" is a matter of perspective. Some might argue that 
> trying to use narrowing at all, for any purpose, is a bad idea, ....

That's plain ludicrous, given the number of couterexamples in Emacs.

> .... and then, of course, the right choice would be to use something
> else.

You seem to want to make widen/narrow-to-region useless for some things.
That is the crux of the matter.  Before making any such change, I'd
rather know what capabilities are being discard in the complexifying of
these primitives.

The right choice for multiple major modes might well be to use something
other than narrowing.  As you know, I proposed something else a few
weeks ago.

> >> How else would you apply each major mode's fontifications only within
> >> its subregions?

> > By adding the appropriate structures as buffer local variables (or
> > perhaps as text properties) which would delimit the subregions, then
> > enhance Font Lock to respect those boundaries.

> Please go ahead and write up this proposal in detail.

No, I was simply answering your question, giving you ideas for further
ideas.

> It doesn't seem as simple as you make it sound. And I wonder if it
> ends up to be different from the "islands" proposal.

Nothing we do here is going to be simple, admittedly.  There might well
be a way of doing things which is a bit like the "islands" proposal but
without its main disadvantage (the huge effort to implement it), and
also a bit like the complexification of narrowing, but without its
disadvantages.

> Further, font-lock is not the only facility we want here. Making 
> syntax-ppss obey region boundaries is another. There's also indentation 
> code, which is more difficult to handle, but limiting visibility there 
> somehow also seems essential.

OK.

> The super mode can also adapt different simpler facilities, like Imenu, 
> using the same approach (go over the chunks, narrow to each, scan using 
> the language-appropriate tools, and concatenate the results).

> >> See
> >> https://github.com/purcell/mmm-mode/blob/c9a857a638701482931ffaaee262b61ce53489f3/mmm-region.el#L789-L816

> > That's rather a lot to take in before breakfast!

> Did you have the breakfast yet? It's just 30 lines.

Ah, I see!  It wasn't clear to me that you were talking just about a
single defun in that file.  I'm not accustomed to reading and
interpreting long URLs.  That was a function which fontifies a number of
regions.  I've forgotton why I should be interested in it.  ;-(

> >> super modes are currently implemented as minor modes. major modes
> >> shouldn't override the choices made by minor modes, if at all possible.

> > What????

> What I said. To the extent the facilities allow, of course.

> > The use/non-use of narrowing is NOT a setting, any more than the
> > use/non-use of cdr is.  They're just general purpose tools.

> Turns out, they are not general purpose enough. That's the problem.

That's an interpretation of the problem.  Your solution appears to be to
make narrowing LESS general, specific to a single problem.

> >> There have been several discussions.

> > They're not coherent, and they're not very useful.

> If there's anything in them that makes you confused, I've yet to hear 
> about it. As far as I can see, you're not interested in making even that 
> effort in learning the subject.

There's nothing in particular that "leaves me confused".  It's that
there no coherent description.  To try and pick everything up from a few
discussions on emacs-devel going arbitrarily far back would take me
several days, and even then I couldn't be sure I'd not missed anything.

> > Could you please write this document.  Given the massive change you want
> > to make in a fundamental Emacs tool, this is not unreasonable to ask.

> I don't enjoy rehashing others' arguments when the original messages are 
> readable enough.

The original messages aren't readable enough for this purpose; some
things are said many times - other things are left out altogether.  The
whole discussion is an incoherent jumble.  That's the nature of mailing
list discussions.  Why do you think I put so much effort into getting
the "islands" proposal coherent - everything said once, and as far as
the flow of the ideas would allow, only once?

What I'm not asking you to rehash is other people's arguments - only the
salient technical points, "one per line".

> It's a small change, by itself. Please read the discussion, starting 
> with http://lists.gnu.org/archive/html/emacs-devel/2016-03/msg01129.html

> And http://lists.gnu.org/archive/html/emacs-devel/2016-03/msg01182.html 
> in particular. After that, please let me know if you have any specific 
> questions.

One of these has a patch, which looks fairly simple.  One thing which
isn't simple is replacing

    (widen)

with

    (let (hard-widen-limits)
      (widen))

in all the places needed.  (I've counted ~705 uses of `widen' in our C
and Lisp sources.)

> > This is, after all, what I did for the "islands" concept a couple of
> > months ago.

> And I have spent considerable time contributing to that discussion as well.

Which is appreciated.

> >> The main thing CC Mode would have to worry about from then on, is that
> >> it won't always be able to goto-char to positions beyond the hard
> >> narrowing, even if they exist in the buffer (and are pointed to by some
> >> buffer-local data structure maintained by CC Mode's parser).

> > That sounds like something worthwhile sorting out in advance.

> Sure. Please go ahead and present your thoughts on the subject.

I can't, as yet.  I don't have enough information.

> I realize it might be rather complicated, actually, to use the narrowing 
> approach with CC Mode as submode. But I don't see any alternative 
> solutions that we're likely to implement. So it may be that some modes 
> will still refuse to work in multi-mode context.

Maybe that design document might, just might, help sort out such
difficulties before they occur.  What I foresee is that these modes
won't flat-out refuse to work with multi major modes, they'll just have
nasty to debug bugs.

And maybe, just maybe, we might somehow get back to discussing the
actual topic of this thread which is a bug concerning syntax-ppss.  One
of the consequences of coupling functions which need not be coupled is
that discussions like this can veer far off topic.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Stefan Monnier]

> perverse, here.  widen and narrow-to-region are enormously useful, and
> it is their simplicity, in large part, which makes them so useful.  You

The problem with them is simple: when a package encounters the buffer
narrowed it has no idea why it's been narrowed, and hence can't know
whether it can/should widen or not, nor how.

Info buffers are the typical example:

- Normally they're narrowed to a "node" and most packages should leave
  the narrowing untouched (including font-lock, and friends) because
  really for all intents and purposes the buffer is meant to only hold
  a node (it's just an implementation detail that in reality the buffer
  holds many nodes and we switch through them via narrowing).

- But of course, if the user comes around and wants to narrow to
  a subpart of a node, she's free to do so (as she should).

- Now, when font-lock comes around and the user has narrowed the buffer,
  font-lock should temporarily widen the buffer back to the whole node
  (but not the whole buffer).

- Similarly, after narrowing to a sub-part, the if the user wants to
  unnnarrow, she'll find that `widen` doesn't do what she expects.
  For this use-case, I could imagine fixing it by providing some kind of
  undo facility for narrowing.


-- Stefan

RE: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Drew Adams]

> after narrowing to a sub-part, the if the user wants to
> unnnarrow, she'll find that `widen` doesn't do what she expects.
> For this use-case, I could imagine fixing it by providing some
> kind of undo facility for narrowing.

Aka this problem: _Narrowing is fine-grained but widening is not._

  Narrowing always narrows to the current region, which can be
  anywhere and have any size up to the buffer size. But command
  'widen' ('C-x n w') always restores the full buffer. There ca
  be many different narrowings (one at a time), but there is
  only one widening: the whole buffer.

One solution:

https://www.emacswiki.org/emacs/MultipleNarrowings

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Dmitry Gutov]

On 06/13/2016 03:28 PM, Alan Mackenzie wrote:

> It does not.  It contains a few relatively vague aspects, undefined
> terms, and "for full details, search!".  I'm not trying to slag off
> Vitalie here, because clearly he was in the middle of trying to figure
> things out, and was unclear about things.

There's no word "search" in the patch.

>> Here's a better link to the message:
>
>> http://lists.gnu.org/archive/html/emacs-devel/2016-03/msg01576.html
>
> I've glanced through that, and I found the doc string you've referred
> to.  It's some help.

Also see the new widen's docstring. It's the whole proposal.

>> If you take a shovel and remove its blade, the result will look very
>> simple. That doesn't mean it's appropriate for any interesting task.
>
> So, you consider widen and narrow-to-region in their current form as
> being as useless as a detached shovel blade?

Maybe a dinner utensil with fork on one side and spoon on the other 
(thus lacking a handle) would be a better illustration.

>> Nowhere does narrow-to-region's documentation say it should only be used
>> in major modes.
>
> Of course not.  As I've said, it's a general purpose tool used
> univerally in Emacs.  I would like to keep it that way.

It wouldn't go anywhere. Its usages, however, would better convey the 
intended use.

>> It's been a known problem for a while, and it has come up in multiple
>> discussions over the last years.
>
> "It's been known"?  To whom?  The only discussion I have seen on the
> topic has been between Stefan, yourself, and Vitalie.  Apologies if I've
> missed anything.

See this discussion, for example: 
https://lists.gnu.org/archive/html/emacs-devel/2015-02/msg01104.html

This subject has come up in virtually every discussion related to mixed 
modes.

> The right choice for multiple major modes might well be to use something
> other than narrowing.  As you know, I proposed something else a few
> weeks ago.

Something we're unlikely to ever get? That's not a good alternative.

>> Please go ahead and write up this proposal in detail.
>
> No, I was simply answering your question, giving you ideas for further
> ideas.

I've been thinking about it for years now, on and off, and changing 
narrowing looks like the best realistic idea so far.

> Nothing we do here is going to be simple, admittedly.  There might well
> be a way of doing things which is a bit like the "islands" proposal but
> without its main disadvantage (the huge effort to implement it), and
> also a bit like the complexification of narrowing, but without its
> disadvantages.

You are welcome to think about and present such a design.

> Ah, I see!  It wasn't clear to me that you were talking just about a
> single defun in that file.  I'm not accustomed to reading and
> interpreting long URLs.  That was a function which fontifies a number of
> regions.  I've forgotton why I should be interested in it.  ;-(

Is it too hard for you to go back a couple of emails are re-read what 
you wrote yourself?

Do you need a recommendation for a better email client?

>>> The use/non-use of narrowing is NOT a setting, any more than the
>>> use/non-use of cdr is.  They're just general purpose tools.
>
>> Turns out, they are not general purpose enough. That's the problem.
>
> That's an interpretation of the problem.  Your solution appears to be to
> make narrowing LESS general, specific to a single problem.

No, it would still serve the existing use cases, just better.

> There's nothing in particular that "leaves me confused".  It's that
> there no coherent description.  To try and pick everything up from a few
> discussions on emacs-devel going arbitrarily far back would take me
> several days, and even then I couldn't be sure I'd not missed anything.

There are about 10-15 short messages in there. That's maybe 20 minutes 
of reading. Certainly less than we've spent on this thread by now.

> The original messages aren't readable enough for this purpose; some
> things are said many times - other things are left out altogether.  The
> whole discussion is an incoherent jumble.

It doesn't look that way to me. And if it does to you, it's quite likely 
that a summary I would present would look just the same.

> Why do you think I put so much effort into getting
> the "islands" proposal coherent - everything said once, and as far as
> the flow of the ideas would allow, only once?

Because you wanted to make your own thoughts clear?

> What I'm not asking you to rehash is other people's arguments - only the
> salient technical points, "one per line".

So that you comment on them, ignoring the previous discussions, and we 
rehash the points that have already been made, yet again?

Let me make one thing clear: it's not my design, and I'm not working on 
that patch. Maybe Vitalie would like to describe the current state of 
the proposal, you should ask him.

> One of these has a patch, which looks fairly simple.  One thing which
> isn't simple is replacing
>
>     (widen)
>
> with
>
>     (let (hard-widen-limits)
>       (widen))
>
> in all the places needed.  (I've counted ~705 uses of `widen' in our C
> and Lisp sources.)

None of those places need to be updated. CC Mode worries about visual 
narrowings, right? It should keep hard-widen-limits in place.

> And maybe, just maybe, we might somehow get back to discussing the
> actual topic of this thread which is a bug concerning syntax-ppss.

Not if you keep forgetting why the discussion went the way it went.

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Alan Mackenzie]

Hello, Dmitry.

On Mon, Jun 13, 2016 at 06:36:56PM +0300, Dmitry Gutov wrote:
> On 06/13/2016 03:28 PM, Alan Mackenzie wrote:

[ .... ]

> > I've glanced through that, and I found the doc string you've referred
> > to.  It's some help.

> Also see the new widen's docstring. It's the whole proposal.

So the way it is to be used would be another proposal.  Given how it would
have the capacity to damage even the user interface, some considerable
care would be required in that use.

> >> If you take a shovel and remove its blade, the result will look very
> >> simple. That doesn't mean it's appropriate for any interesting task.

> > So, you consider widen and narrow-to-region in their current form as
> > being as useless as a detached shovel blade?

> Maybe a dinner utensil with fork on one side and spoon on the other 
> (thus lacking a handle) would be a better illustration.

> >> Nowhere does narrow-to-region's documentation say it should only be used
> >> in major modes.

> > Of course not.  As I've said, it's a general purpose tool used
> > univerally in Emacs.  I would like to keep it that way.

> It wouldn't go anywhere. Its usages, however, would better convey the 
> intended use.

Its uses are currently unconstrained.  I don't know what you mean by
"convey the intended use": the intended use is to restrict the accessible
portion of the buffer.

> >> It's been a known problem for a while, and it has come up in multiple
> >> discussions over the last years.

> > "It's been known"?  To whom?  The only discussion I have seen on the
> > topic has been between Stefan, yourself, and Vitalie.  Apologies if I've
> > missed anything.

> See this discussion, for example: 
> https://lists.gnu.org/archive/html/emacs-devel/2015-02/msg01104.html

> This subject has come up in virtually every discussion related to mixed 
> modes.

OK.  One tentative conclusion from that is that narrowing is not the
ideal core concept for implementing multiple major modes.  I assert that
new core facilities are needed to implement these properly.

[ .... ]

> I've been thinking about it for years now, on and off, and changing 
> narrowing looks like the best realistic idea so far.

> > Nothing we do here is going to be simple, admittedly.  There might well
> > be a way of doing things which is a bit like the "islands" proposal but
> > without its main disadvantage (the huge effort to implement it), and
> > also a bit like the complexification of narrowing, but without its
> > disadvantages.

> You are welcome to think about and present such a design.

I've already presented one!  The implementation of that is not outside
the bounds of possibilities, but it would need more people than just me
working on it.  So far nobody else seems enthused enough.

> > Ah, I see!  It wasn't clear to me that you were talking just about a
> > single defun in that file.  I'm not accustomed to reading and
> > interpreting long URLs.  That was a function which fontifies a number of
> > regions.  I've forgotton why I should be interested in it.  ;-(

> Is it too hard for you to go back a couple of emails are re-read what 
> you wrote yourself?

Not just once, no.  But continually having to jump to a web client to
chase up old emacs-devel discussions, and continually having to go
backwards in the current thread (because the necessary context has
already been snipped from the current post) makes things very tedious.

This is like why some code is difficult to understand and follow.  It
happens because of the need continually to look somewhere else.

> Do you need a recommendation for a better email client?

No.

[ .... ]

> > There's nothing in particular that "leaves me confused".  It's that
> > there no coherent description.  To try and pick everything up from a few
> > discussions on emacs-devel going arbitrarily far back would take me
> > several days, and even then I couldn't be sure I'd not missed anything.

> There are about 10-15 short messages in there. That's maybe 20 minutes 
> of reading. Certainly less than we've spent on this thread by now.

In where?  The context has been snipped already.  It might be 20 minutes
of reading.  And then several hours trying to tie up all the loose ends,
trying to work out what the participants have been taking for granted,
tying the disjointed pieces together into some sort of whole.

> > The original messages aren't readable enough for this purpose; some
> > things are said many times - other things are left out altogether.  The
> > whole discussion is an incoherent jumble.

> It doesn't look that way to me. And if it does to you, it's quite likely 
> that a summary I would present would look just the same.

Not at all.  You're quite a good writer.  ;-)

> > Why do you think I put so much effort into getting
> > the "islands" proposal coherent - everything said once, and as far as
> > the flow of the ideas would allow, only once?

> Because you wanted to make your own thoughts clear?

I wanted to stimulate fruitful discussion.  This actually happened to a
significant extent.

> > What I'm not asking you to rehash is other people's arguments - only the
> > salient technical points, "one per line".

> So that you comment on them, ignoring the previous discussions, and we 
> rehash the points that have already been made, yet again?

Yes, of course.

> Let me make one thing clear: it's not my design, and I'm not working on 
> that patch. Maybe Vitalie would like to describe the current state of 
> the proposal, you should ask him.

I'll wait for Vitalie to reappear.

> > One of these has a patch, which looks fairly simple.  One thing which
> > isn't simple is replacing

> >     (widen)

> > with

> >     (let (hard-widen-limits)
> >       (widen))

> > in all the places needed.  (I've counted ~705 uses of `widen' in our C
> > and Lisp sources.)

> None of those places need to be updated.

That's a very "brave" thing to say.  Picture a function which saves a
buffer's contents, and which first widens to get the buffer's
boundaries....

> CC Mode worries about visual narrowings, right? It should keep
> hard-widen-limits in place.

I don't know what you mean by "visual narrowings".  CC Mode uses
narrow-to-region and widen, full stop, as do its users.  Whether it
leaves hard-widen-limits in place depends on how they have been set.
Which all comes back to the current uncertainty surrounding the proposal.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 426 bytes --]

>>>>> Dmitry Gutov <dgutov@yandex.ru> writes:

> I'm still not even clear which branch 25.2 will come from, and that's an
> important concern.

It will be created by cherry-picking from master, and from changes made
directly to emacs-25 after the release of 25.1.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 651 bytes --]

>>>>> Alan Mackenzie <acm@muc.de> writes:

> I think I'd be happy enough about it being fixed in master, if the fix
> happens before the release. :-). The trouble is, if there's no deadline,
> it'll never get fixed.

Hi Alan,

I agree that this is a wart that needs fixing, but there is no reason to
assign a particular timeline to it above our other issues. Since it's not tied
to a specific release, it will be fixed whenever people get around to fixing
it, as with any open issue.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Stefan Monnier]

> My suggestion for a fix, discussed in the thread for the bug, is to come
> up with a new function similar to syntax-ppss which always returns the
> equivalent of
>     (parse-partial-sexp 1 pos)
> in place of syntax-ppss's
>     (parse-partial-sexp (point-min) pos)

Almost agreed, except that something should be able to control the "1".

Most/all users of `syntax-ppss' indeed want to start "from the
beginning", but in some cases "the beginning" they want isn't 1 (hence
things like font-lock-dont-widen).

Ideally the "control" should not be specific to syntax.el and should
make font-lock-dont-widen obsolete.  Vitalie Spinu was working on such
a patch.


        Stefan

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Alan Mackenzie]

Hello, Stefan.

On Wed, Jun 08, 2016 at 08:43:43AM -0400, Stefan Monnier wrote:
> > My suggestion for a fix, discussed in the thread for the bug, is to come
> > up with a new function similar to syntax-ppss which always returns the
> > equivalent of
> >     (parse-partial-sexp 1 pos)
> > in place of syntax-ppss's
> >     (parse-partial-sexp (point-min) pos)

> Almost agreed, except that something should be able to control the "1".

You mean, maybe something like `syntax-ppss-base' which, if non-nil,
would be the "1".  Various bits of software (mainly super modes) could
then set or bind this to non-nil values.

> Most/all users of `syntax-ppss' indeed want to start "from the
> beginning", but in some cases "the beginning" they want isn't 1 (hence
> things like font-lock-dont-widen).

font-lock-dont-widen is ugly indeed.  Maybe syntax-ppss-base could be a
bit less ugly.

> Ideally the "control" should not be specific to syntax.el and should
> make font-lock-dont-widen obsolete.  Vitalie Spinu was working on such
> a patch.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Stefan Monnier]

> You mean, maybe something like `syntax-ppss-base' which, if non-nil,
> would be the "1".  Various bits of software (mainly super modes) could
> then set or bind this to non-nil values.

Yes, could be something like that.


        Stefan

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Dmitry Gutov]

On 06/11/2016 06:00 PM, Stefan Monnier wrote:
>> You mean, maybe something like `syntax-ppss-base' which, if non-nil,
>> would be the "1".  Various bits of software (mainly super modes) could
>> then set or bind this to non-nil values.
>
> Yes, could be something like that.

If we follow this path, and not hard-widen, the proposal must carefully 
consider the existing use cases for font-lock-dont-widen. Because if 
syntax-ppss always widens, even if it uses a different base for the 
state computation, it will undo the effect of font-lock-dont-widen 
(because font-lock-fontify-syntactically-region calls syntax-propertize, 
and that calls syntax-ppss), at least to some extent.

Can we be sure that font-lock-fontify-keywords-region stays within the 
given limits without narrowing?

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Stefan Monnier]

> If we follow this path, and not hard-widen, the proposal must carefully
> consider the existing use cases for font-lock-dont-widen. Because if

Yes, the new mechanism should really supercede font-lock-dont-widen.

Just syntax-ppss-base as a single integer probably wouldn't cut it.
It should probably be a setting that's not specifically tied to
`syntax'.
And as you point out in another message, it might make sense to have it
specify an upper-bound too.


        Stefan

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Alan Mackenzie]

Hello, Stefan and Dmitry.

On Sat, Jun 11, 2016 at 05:24:25PM -0400, Stefan Monnier wrote:
> > If we follow this path, and not hard-widen, the proposal must carefully
> > consider the existing use cases for font-lock-dont-widen. Because if

Er, hang on a moment, you two!  The thread is supposed to be about
syntax-ppss.  syntax-ppss is supposed to be a function which does one
thing and does it well.

That one thing is to determine, possibly from a cache, the equivalent to

    (parse-partial-sexp "1" pos)

, where "1" may take non-canonical values.

Use cases for font-lock-dont-widen, existing or not, HAVE NO BEARING on
that determination of the parse-partial-sexp equivalent.  So why are we
talking about them here?

> Yes, the new mechanism should really supercede font-lock-dont-widen.

> Just syntax-ppss-base as a single integer probably wouldn't cut it.
> It should probably be a setting that's not specifically tied to
> `syntax'.

How can anything more than a single integer be required to specify the
desired value of "1"?

> And as you point out in another message, it might make sense to have it
> specify an upper-bound too.

Maybe, but that's got nothing to do with syntax-ppss.

Can we PLEASE not confuse the specification and inner workings of
syntax-ppss with the contexts in which it may or may not be used?

I propose one third of us should now fix syntax-ppss so that it conforms
to its (new) specification, and that syntax-ppss-base should comprise an
integral part of this fix.

I don't consider myself to be the best of the three of us for this job
(given that I dislike syntax-ppss) but I'll do it if nobody else is
willing to.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Stefan Monnier]

> That one thing is to determine, possibly from a cache, the equivalent to

>     (parse-partial-sexp "1" pos)

> , where "1" may take non-canonical values.

> Use cases for font-lock-dont-widen, existing or not, HAVE NO BEARING on
> that determination of the parse-partial-sexp equivalent.  So why are we
> talking about them here?

We're talking about them because the cases where font-lock-dont-widen is
needed are conceptually the same as those where syntax-ppss would need
something else than 1, so we'd like to come up with something that can
cover both uses at the same time, so as to have a single setting rather
than code that sets font-lock-dont-widen, jit-lock-dont-widen,
syntax-ppss-base, widen-user-limit, and whatnot.


        Stefan

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Alan Mackenzie]

Hello, Stefan.

On Sat, Jun 11, 2016 at 05:49:59PM -0400, Stefan Monnier wrote:
> > That one thing is to determine, possibly from a cache, the equivalent to

> >     (parse-partial-sexp "1" pos)

> > , where "1" may take non-canonical values.

> > Use cases for font-lock-dont-widen, existing or not, HAVE NO BEARING on
> > that determination of the parse-partial-sexp equivalent.  So why are we
> > talking about them here?

> We're talking about them because the cases where font-lock-dont-widen is
> needed are conceptually the same as those where syntax-ppss would need
> something else than 1, .....

_Some_ of the use cases might exhibit this conceptual similarity.  Some
won't.

Can we please not mix up the use cases with the specification?

> .... so we'd like to come up with something that can cover both uses
> at the same time, so as to have a single setting rather than code that
> sets font-lock-dont-widen, jit-lock-dont-widen, syntax-ppss-base,
> widen-user-limit, and whatnot.

That's entirely the Wrong Thing to do.  We want syntax-ppss not to have
ugly couplings with other functions.  If, as a programmer, I want, say,
to set "1" to 1, I want to be able to do this without fouling up the
functionality of all these other things.  Otherwise the function is less
useful than it should be.

If it turns out in some circumstances that several variables have the
same value, we can set each of them.  It's no big deal.  We could even
use a macro to do it.

Yet again, can we please agree on a strong coherent specification for
syntax-ppss, independent of its use cases, and get that coded up?

Thanks!

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Stefan Monnier]

>> We're talking about them because the cases where font-lock-dont-widen is
>> needed are conceptually the same as those where syntax-ppss would need
>> something else than 1, .....
> _Some_ of the use cases might exhibit this conceptual similarity.
> Some won't.

Can't think of any that won't.  Some may not care (e.g. because they
only use syntax-ppss and not font-lock), but I can't think of any where
we'd need/want to specify different values for the various packages that
could use such info.

> That's entirely the Wrong Thing to do.  We want syntax-ppss not to have
> ugly couplings with other functions.

I look at it from the exact opposite viewpoint: the packages that need
to set those boundaries may not even need/want to use syntax-ppss (or
font-lock) themselves, so it'd be completely wrong for them to have to
set a syntax-ppss-* variable (or a font-lock-* variable).


        Stefan

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Dmitry Gutov]

On 06/12/2016 12:49 AM, Stefan Monnier wrote:

> We're talking about them because the cases where font-lock-dont-widen is
> needed are conceptually the same as those where syntax-ppss would need
> something else than 1,

And also because the straightforward implementation of Alan's proposal 
for syntax-ppss to widen and then maybe apply syntax-ppss-base (if it's 
non-nil) will break font-lock-dont-widen. Like I've explained in another 
message.

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Dmitry Gutov]

On 06/12/2016 12:44 AM, Alan Mackenzie wrote:

> Use cases for font-lock-dont-widen, existing or not, HAVE NO BEARING on
> that determination of the parse-partial-sexp equivalent.  So why are we
> talking about them here?

Because `widen' is viral: if any callees of the current function call 
it, the current narrowing is busted. If you don't want to support 
narrowing in syntax-ppss, we can't support it in font-lock either.

> I propose one third of us should now fix syntax-ppss so that it conforms
> to its (new) specification, and that syntax-ppss-base should comprise an
> integral part of this fix.

Please read the other messages.

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Dmitry Gutov]

On 06/11/2016 01:24 PM, Alan Mackenzie wrote:

> font-lock-dont-widen is ugly indeed.  Maybe syntax-ppss-base could be a
> bit less ugly.

The latter can't be the full replacement for the former:

a) font-lock-dont-widen is used well before syntax-ppss-base would take 
effect. In particular, it affects the result of 
font-lock-extend-region-functions.

b) font-lock-dont-widen helps with both upper and lower bounds on the 
buffer position. syntax-ppss-base would only describe one.

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Alan Mackenzie]

Hello, Dmitry.

On Sat, Jun 11, 2016 at 07:32:44PM +0300, Dmitry Gutov wrote:
> On 06/11/2016 01:24 PM, Alan Mackenzie wrote:

> > font-lock-dont-widen is ugly indeed.  Maybe syntax-ppss-base could be a
> > bit less ugly.

> The latter can't be the full replacement for the former:

> a) font-lock-dont-widen is used well before syntax-ppss-base would take 
> effect. In particular, it affects the result of 
> font-lock-extend-region-functions.

> b) font-lock-dont-widen helps with both upper and lower bounds on the 
> buffer position. syntax-ppss-base would only describe one.

I'm fighting blindfolded with my hands tied behind my back here.

My gut feeling is that there's something wrong with the whole idea.

Where is the documentation for the font-lock-dont-widen (etc.)
mechanism?  It has to exist.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Dmitry Gutov]

On 06/11/2016 10:58 PM, Alan Mackenzie wrote:

> My gut feeling is that there's something wrong with the whole idea.

That often happens when a programmer stumbles on a problem of unexpected 
complexity.

> Where is the documentation for the font-lock-dont-widen (etc.)
> mechanism?  It has to exist.

I don't think anybody has written a manual entry for that. Especially 
not for its uses in third-party code.

First, consider what I've already written in this thread.

Second, the hard-widen thread has some discussion of the code inside 
Emacs using font-lock-dont-widen (and how a well-designed hard-widen 
could solve that problem, too). E.g., I see a reference to it in 
rmail-variables.

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Andreas Röhler]

On 11.06.2016 21:58, Alan Mackenzie wrote:
> I'm fighting blindfolded with my hands tied behind my back here. My 
> gut feeling is that there's something wrong with the whole idea. Where 
> is the documentation for the font-lock-dont-widen (etc.) mechanism? It 
> has to exist. 

Hi Alan,

the whole thing needs to be reflected from bottom. Not tied to 
exceptions like font-lock-dont-widen.

Why syntax-ppss might be useful at all? You told me the case lately: it 
provides a cache.

Okay. But when we really need the cache?

The solutions IMHO is first to restrict all usage of syntax-ppss to real 
necessity,  employ parse-partial-sexp as far as possible instead. Once 
syntax-ppss is reduced and restricted, fixing and tailoring it should be 
easier.

Cheers,

Andreas

Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Stefan Monnier]

> Okay. But when we really need the cache?

Won't make a difference.  The same problems discussed here would show up
if syntax-ppss didn't use a cache (some other problems wouldn't,
admittedly, such as those linked to using syntax-ppss from within
with-syntax-table).

Please please pretty please.  Try to understand syntax-ppss before
commenting on it.


        Stefan