Syntax ambiguities in narrowed buffers and multiple major modes: a proposed solution.

Syntax ambiguities in narrowed buffers and multiple major modes: a proposed solution.

[Alan Mackenzie]

Hello, Emacs.

This post proposes a method of coding up syntactic "islands" as a
solution to various syntactic problems in the current Emacs.  It aims to
address the various problems raised in the thread Subject: Bug #25608
and the comment-cache branch.

(i) Problems:
  o - Ambiguity involved with narrowed regions - sometimes programs and
    users wish to see syntactic entities (e.g. strings, comments) as
    though point-min were a syntactically neutral position - other times
    they want the syntax to be relative to the beginning of the buffer.
  o - Multiple major modes - each submode region should start
    syntactically at a syntactically neutral position, regardless of the
    syntactic context of the enclosing supermode.  This region should be
    able to have its own syntax table.

(ii) Proposed solution - islands.
  o - By default, in a narrowed region, the syntactic context of a
    position will be that relative to BOB.
  o - There will be two new syntax classes introduced for use in syntax
    table text properties: "island open" and "island close".  Together,
    these enclose an "island", a region of the buffer syntactically
    disjoint from the text outside of the region.
  o - An island open syntax class will have, in place of the matching
    character argument (used by open/close parenthesis classes), a list
    argument containing:
    * - the (optional) syntax table for the island;
    * - the "previous syntax" of the position, which has been overwritten
      by the island open class syntax-table text property.
    * - Possible further items.
  o - An island close syntax class will have, in place of the
    matching character argument, a list containing:
    * - The "previous syntax" of the position.
    * - Possible further items.

(iii) Effect on parse-partial-sexp/syntax-ppss.
  o - The 10-element parser state would become a stack of parser states.
  o - Each time parse-partial-sexp encounters an island open, it pushes
    the current state, including the syntax table, onto this stack,
    reinitialises the state, and optionally sets the syntax table, for
    the island just being opened.  The island begins at the buffer
    position _after_ the one bearing the island open syntax.
  o - Each time parse-partial-sexp encounters an island close, it pops
    the suspended state, including the syntax table, off the stack of
    states.  (If there is no stacked state, the function stops at this
    point, somewhat analagously to reaching the end of buffer.)
  o - parse-partial-sexp would be amended to allow the caller to direct
    the parsing to stop at the end (and possibly beginning) of an island,
    much as it currently does for strings and comments.

(iv) Modification of certain commands and functions.
  o - narrow-to-region will be given an optional argument which, if set,
    directs Emacs to make the new region an island.  Thus, C-u C-x n n
    would enable a user to narrow to a "comment within a string" and edit
    it as though it were a comment.
  o - A new function would be needed to get the "base" syntax of a
    position, the syntax it would have if it weren't for any island
    open/close syntax table properties on it.
  o - forward-comment will need to cope, one way or another, with the
    possibility of an island inside a comment.  Several other scanning
    commands will likewise need amendment.

(v) Advantages of this proposal.
  o - The ambiguity in narrowed regions would be resolved without loss of
    functionality.
  o - The fact that the syntactic context is always global in a buffer
    facilitates the building and manipulation of syntax caches (of
    whatever nature).
  o - Multiple major mode modes could be implemented without abusing the
    narrowing mechanism.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Syntax ambiguities in narrowed buffers and multiple major modes: a proposed solution.

[Stefan Monnier]

Sounds fairly close to some of the ideas I toyed with.
Here are a few comments:

>   o - Ambiguity involved with narrowed regions - sometimes programs and
>     users wish to see syntactic entities (e.g. strings, comments) as
>     though point-min were a syntactically neutral position - other times
>     they want the syntax to be relative to the beginning of the buffer.

I don't think this need is very serious for users.
So, I think we should focus on those cases where this is used by Elisp code.

>   o - There will be two new syntax classes introduced for use in syntax
>     table text properties: "island open" and "island close".  Together,
>     these enclose an "island", a region of the buffer syntactically
>     disjoint from the text outside of the region.

[ I like to consider that strings and comments are also a form of
  "island", although we're probably better off supporting them in
  a special way like we do now.  ]

I think we should try not to limit ourselves to nesting of islands.
IOW, we should strive to find a design where a single char can close an
island and open another one.

>   o - narrow-to-region will be given an optional argument which, if set,
>     directs Emacs to make the new region an island.  Thus, C-u C-x n n
>     would enable a user to narrow to a "comment within a string" and edit
>     it as though it were a comment.

How would this work (especially for uses from Elisp)?
Would it set syntax-table text-properties?


        Stefan

Re: Syntax ambiguities in narrowed buffers and multiple major modes: a proposed solution.

[Alan Mackenzie]

Hello, Stefan

On Sat, Feb 25, 2017 at 14:42:06 -0500, Stefan Monnier wrote:
> Sounds fairly close to some of the ideas I toyed with.

Excellent!

> Here are a few comments:

> >   o - Ambiguity involved with narrowed regions - sometimes programs and
> >     users wish to see syntactic entities (e.g. strings, comments) as
> >     though point-min were a syntactically neutral position - other times
> >     they want the syntax to be relative to the beginning of the buffer.

> I don't think this need is very serious for users.
> So, I think we should focus on those cases where this is used by Elisp
> code.

I don't really see the distinction between users and code here.  If we
implement for one, it will work for the other, won't it?

> >   o - There will be two new syntax classes introduced for use in syntax
> >     table text properties: "island open" and "island close".  Together,
> >     these enclose an "island", a region of the buffer syntactically
> >     disjoint from the text outside of the region.

> [ I like to consider that strings and comments are also a form of
>   "island", although we're probably better off supporting them in
>   a special way like we do now.  ]

I think that's just confusing the meaning of "island", which I'd like to
keep clear and unambiguous.  Something to be decided is how we'd handle
an island within a comment or string.

> I think we should try not to limit ourselves to nesting of islands.
> IOW, we should strive to find a design where a single char can close an
> island and open another one.

That would need just a minor extension of what I set out, I think.
Something like an "island swap" syntax class.  This might find use in
languages like lex and yacc.

> >   o - narrow-to-region will be given an optional argument which, if set,
> >     directs Emacs to make the new region an island.  Thus, C-u C-x n n
> >     would enable a user to narrow to a "comment within a string" and edit
> >     it as though it were a comment.

> How would this work (especially for uses from Elisp)?
> Would it set syntax-table text-properties?

Yes, it would.  It would put an island open syntax-table property on the
character before START, and and island close on the character after END.
This would isolate the region syntactically from its surroudings.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Syntax ambiguities in narrowed buffers and multiple major modes: a proposed solution.

[Stefan Monnier]

> I don't really see the distinction between users and code here.

I think the details are very different: in Elisp code, it's typically
combined with save-restriction, it's short lived, and performance is
fairly important.  For C-x n n none of those three aspects apply.

> If we implement for one, it will work for the other, won't it?

It's quite likely that if we can make it work for Elisp, we can also
make it work satisfactorily for C-x n n.  But the other way around is
not necessarily true, I think.

>> [ I like to consider that strings and comments are also a form of
>> "island", although we're probably better off supporting them in
>> a special way like we do now.  ]
> I think that's just confusing the meaning of "island", which I'd like to
> keep clear and unambiguous.  Something to be decided is how we'd handle
> an island within a comment or string.

Suit yourself.  I find it to be a good way to think about it.  I don't
see why an island within a comment/string should need any special
treatment.  Just like an island within an island.

>> >   o - narrow-to-region will be given an optional argument which, if set,
>> >     directs Emacs to make the new region an island.  Thus, C-u C-x n n
>> >     would enable a user to narrow to a "comment within a string" and edit
>> >     it as though it were a comment.
>> How would this work (especially for uses from Elisp)?
>> Would it set syntax-table text-properties?
> Yes, it would.  It would put an island open syntax-table property on the
> character before START, and and island close on the character after END.
> This would isolate the region syntactically from its surroudings.

I don't think that's going to be fast enough, then.  I'm thinking of
cases where current Elisp code does something like

    (save-restriction
      (narrow-to-region ...)
      (with-syntax-table ...
        (backward-sexp 1)))

in order to efficiently jump over a small element (e.g. an SGML tag) and
may very well want to do this within a loop.

This usage doesn't correspond to an island, really and shouldn't cause
caches to be flushed.


        Stefan

Re: Syntax ambiguities in narrowed buffers and multiple major modes: a proposed solution.

[Alan Mackenzie]

Hello, Stefan.

On Sat, Feb 25, 2017 at 21:32:49 -0500, Stefan Monnier wrote:
> > I don't really see the distinction between users and code here.

> I think the details are very different: in Elisp code, it's typically
> combined with save-restriction, it's short lived, and performance is
> fairly important.  For C-x n n none of those three aspects apply.

Sorry, I've lost the thread, here.  The original point was that there is
currently an ambiguity in narrowed regions - that sometimes the
code/user wants to treat point-min as a syntactically neutral point,
other times it wants beginning of buffer to be that neutral point.

I think you've moved onto talking about something else, without saying
exactly what that something else is.

> > If we implement for one, it will work for the other, won't it?

> It's quite likely that if we can make it ...

"it" has no referent.  What is "it"?

> .... work for Elisp, we can also make it work satisfactorily for C-x n
> n.  But the other way around is not necessarily true, I think.

> >> [ I like to consider that strings and comments are also a form of
> >> "island", although we're probably better off supporting them in
> >> a special way like we do now.  ]

> > I think that's just confusing the meaning of "island", which I'd like to
> > keep clear and unambiguous.  Something to be decided is how we'd handle
> > an island within a comment or string.

> Suit yourself.  I find it to be a good way to think about it.

In that case, we'd need some other term to mean what I'm calling an
"island", i.e. a region of buffer bounded by island open/close
syntax-table text properties, possibly with its own syntax table, which
is syntactically disjoint from the surrounding buffer pieces.

> I don't see why an island within a comment/string should need any
> special treatment.  Just like an island within an island.

It doesn't need special treatment, but it does need to be dealt with
somehow.  Probably by silently skipping over the island.  A bit like how
comments are usually skipped in scan-lists.  But I think you're right,
it's not a big thing.

> >> >   o - narrow-to-region will be given an optional argument which, if set,
> >> >     directs Emacs to make the new region an island.  Thus, C-u C-x n n
> >> >     would enable a user to narrow to a "comment within a string" and edit
> >> >     it as though it were a comment.

> >> How would this work (especially for uses from Elisp)?
> >> Would it set syntax-table text-properties?

> > Yes, it would.  It would put an island open syntax-table property on the
> > character before START, and and island close on the character after END.
> > This would isolate the region syntactically from its surroudings.

> I don't think that's going to be fast enough, then.  I'm thinking of
> cases where current Elisp code does something like

>     (save-restriction
>       (narrow-to-region ...)
>       (with-syntax-table ...
>         (backward-sexp 1)))

> in order to efficiently jump over a small element (e.g. an SGML tag) and
> may very well want to do this within a loop.

Is there any need in that example for the narrow-to-region call to
create an island[*]?  Or, more precisely, _when_ is there any need to
create an island?  If this did need an island and were within a loop,
surely the code could be rearranged for the loop to be inside the
with-syntax-table form.

[*] I'm envisaging narrow-to-region getting an &optional parameter
make-island, if that's not clear.

> This usage doesn't correspond to an island, really and shouldn't cause
> caches to be flushed.

I don't think that code would normally need an island.  But the caches
(in particular, the syntax-ppss cache) are invalid inside the
with-syntax-table form anyway, and in the general case that has to be
dealt with somehow.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Syntax ambiguities in narrowed buffers and multiple major modes: a proposed solution.

[Yuri Khan]

On Sun, Feb 26, 2017 at 7:06 PM, Alan Mackenzie <acm@muc.de> wrote:

> In that case, we'd need some other term to mean what I'm calling an
> "island", i.e. a region of buffer bounded by island open/close
> syntax-table text properties, possibly with its own syntax table, which
> is syntactically disjoint from the surrounding buffer pieces.

In many languages, conventions exist for including formal
documentation in the source. In Elisp and Python, it is syntactically
expressed as a string. In Javadoc and Doxygen (used for C and C++), it
takes the form of a specially formatted comment. In C#, it is a
comment containing XML markup.

In all cases, the documentation has its own syntax, distinct from the
syntax of the surrounding program, and may benefit from local
bindings. Totally a use case for islands.

Re: Syntax ambiguities in narrowed buffers and multiple major modes: a proposed solution.

[Alan Mackenzie]

Hello, Yuri.

On Sun, Feb 26, 2017 at 19:24:41 +0700, Yuri Khan wrote:
> On Sun, Feb 26, 2017 at 7:06 PM, Alan Mackenzie <acm@muc.de> wrote:

> > In that case, we'd need some other term to mean what I'm calling an
> > "island", i.e. a region of buffer bounded by island open/close
> > syntax-table text properties, possibly with its own syntax table, which
> > is syntactically disjoint from the surrounding buffer pieces.

> In many languages, conventions exist for including formal
> documentation in the source. In Elisp and Python, it is syntactically
> expressed as a string. In Javadoc and Doxygen (used for C and C++), it
> takes the form of a specially formatted comment. In C#, it is a
> comment containing XML markup.

> In all cases, the documentation has its own syntax, distinct from the
> syntax of the surrounding program, and may benefit from local
> bindings. Totally a use case for islands.

Could well be.  Once the facility has been implemented, people will be
able to try out things like this.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Syntax ambiguities in narrowed buffers and multiple major modes: a proposed solution.

[Stefan Monnier]

>> > I don't really see the distinction between users and code here.
>> I think the details are very different: in Elisp code, it's typically
>> combined with save-restriction, it's short lived, and performance is
>> fairly important.  For C-x n n none of those three aspects apply.
> Sorry, I've lost the thread, here.  The original point was that there is
> currently an ambiguity in narrowed regions - that sometimes the
> code/user wants to treat point-min as a syntactically neutral point,
> other times it wants beginning of buffer to be that neutral point.
> I think you've moved onto talking about something else, without saying
> exactly what that something else is.

Hmm... no, that's exactly what I'm talking about.  I'm pointing out that
cases where Elisp code uses narrowing is technically quite different
from cases where `C-x n n` is used.  In both cases, there is an
ambiguity (i.e. some uses expect syntax to start at point-min others at
1).

>> > If we implement for one, it will work for the other, won't it?
>> It's quite likely that if we can make it ...
> "it" has no referent.  What is "it"?

The same as your "it", AFAIK (i.e. "it will work" meaning something like
"we provide a way for the user/code to tell Emacs where syntax starts").

>> Suit yourself.  I find it to be a good way to think about it.
> In that case, we'd need some other term to mean what I'm calling an
> "island", i.e. a region of buffer bounded by island open/close
> syntax-table text properties, possibly with its own syntax table, which
> is syntactically disjoint from the surrounding buffer pieces.

No, as I said, it's just a way to think about the *problem*.  In the
actual solution/API/implementation we'l probably still want to treat
strings/comments specially rather than as islands.

>> (save-restriction
>>   (narrow-to-region ...)
>>   (with-syntax-table ...
>>     (backward-sexp 1)))
>> in order to efficiently jump over a small element (e.g. an SGML tag) and
>> may very well want to do this within a loop.
> Is there any need in that example for the narrow-to-region call to
> create an island[*]?

"create"?  As such, no.
But the issue is that the syntax beginning in the above example should be
point-min, not 1.  AFAICT in your currently suggested solution you have
no other way to get that behavior than to "create" an island.

BTW, I'm quite willing to tell authors that the above chunk of code
needs to be rewritten with a new macro, if that can help.

> I don't think that code would normally need an island.  But the caches
> (in particular, the syntax-ppss cache) are invalid inside the
> with-syntax-table form anyway, and in the general case that has to be
> dealt with somehow.

Right.  But I think we need to resolve this "somehow" as part of the
new "island" design.


        Stefan

Re: Syntax ambiguities in narrowed buffers and multiple major modes: a proposed solution.

[Alan Mackenzie]

Hello, Stefan.

On Sun, Feb 26, 2017 at 08:47:55 -0500, Stefan Monnier wrote:
> >> > I don't really see the distinction between users and code here.

> >> I think the details are very different: in Elisp code, it's typically
> >> combined with save-restriction, it's short lived, and performance is
> >> fairly important.  For C-x n n none of those three aspects apply.

> > Sorry, I've lost the thread, here.  The original point was that there is
> > currently an ambiguity in narrowed regions - that sometimes the
> > code/user wants to treat point-min as a syntactically neutral point,
> > other times it wants beginning of buffer to be that neutral point.
> > I think you've moved onto talking about something else, without saying
> > exactly what that something else is.

> Hmm... no, that's exactly what I'm talking about.  I'm pointing out that
> cases where Elisp code uses narrowing is technically quite different
> from cases where `C-x n n` is used.  In both cases, there is an
> ambiguity (i.e. some uses expect syntax to start at point-min others at
> 1).

OK.  Resolving that ambiguity is what my proposal will do.

> >> > If we implement for one, it will work for the other, won't it?
> >> It's quite likely that if we can make it ...
> > "it" has no referent.  What is "it"?

> The same as your "it", AFAIK (i.e. "it will work" meaning something like
> "we provide a way for the user/code to tell Emacs where syntax starts").

> >> Suit yourself.  I find it to be a good way to think about it.
> > In that case, we'd need some other term to mean what I'm calling an
> > "island", i.e. a region of buffer bounded by island open/close
> > syntax-table text properties, possibly with its own syntax table, which
> > is syntactically disjoint from the surrounding buffer pieces.

> No, as I said, it's just a way to think about the *problem*.  In the
> actual solution/API/implementation we'l probably still want to treat
> strings/comments specially rather than as islands.

I am proposing implementing islands as a solution to the problem, not as
a way of thinking about it.

> >> (save-restriction
> >>   (narrow-to-region ...)
> >>   (with-syntax-table ...
> >>     (backward-sexp 1)))
> >> in order to efficiently jump over a small element (e.g. an SGML tag) and
> >> may very well want to do this within a loop.

> > Is there any need in that example for the narrow-to-region call to
> > create an island[*]?

> "create"?  As such, no.
> But the issue is that the syntax beginning in the above example should be
> point-min, not 1.

Should it?  Why?  Are you positing this as a possibility, or are you
saying that in this code, the syntax beginning must always definitely
begin at point-min.

When does it actually make a difference, apart from when point-min is
inside a string or a comment?  I don't think it makes a difference in
the (backward-sexp 1) case above.

> AFAICT in your currently suggested solution you have no other way to
> get that behavior than to "create" an island.

That is correct.  The creation of an island will not be an expensive
action, unless it is in a tight loop - recording two current syntactic
elements, and calculating and setting two syntax-table text properties;
then restoring these later.

> BTW, I'm quite willing to tell authors that the above chunk of code
> needs to be rewritten with a new macro, if that can help.

Thanks.

> > I don't think that code would normally need an island.  But the caches
> > (in particular, the syntax-ppss cache) are invalid inside the
> > with-syntax-table form anyway, and in the general case that has to be
> > dealt with somehow.

> Right.  But I think we need to resolve this "somehow" as part of the
> new "island" design.

Do we?  This is more a problem with syntax-ppss, which needs resolving
regardless of whether or not islands get implemented.

In the case where an island is a permanent feature of a buffer with its
own syntax table, the syntax-ppss cache would not become invalid on
scanning past the island open position.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Syntax ambiguities in narrowed buffers and multiple major modes: a proposed solution.

[Stefan Monnier]

> I am proposing implementing islands as a solution to the problem, not as
> a way of thinking about it.

In order to design a solution, we need to think about the problem somehow.

>> But the issue is that the syntax beginning in the above example should be
>> point-min, not 1.
> Should it?

It's not really inherent in the code, but it's how it currently behaves
(mostly), and in some cases that is what the author wants.  In other
cases, the author wants something else.

> When does it actually make a difference, apart from when point-min is
> inside a string or a comment?  I don't think it makes a difference in
> the (backward-sexp 1) case above.

backward-sexp will call back_comment, so yes it makes a difference.
And since speed may be important, it's also important to make sure we
don't scan over the 100KB of text that comes before point-min.

> That is correct.  The creation of an island will not be an expensive
> action, unless it is in a tight loop - recording two current syntactic
> elements, and calculating and setting two syntax-table text properties;
> then restoring these later.

But won't setting those islands flush the caches (e.g. comment-cache,
syntax-ppss cache, ...)?


        Stefan

Re: Syntax ambiguities in narrowed buffers and multiple major modes: a proposed solution.

[Alan Mackenzie]

Hello, Stefan.

On Sun, Feb 26, 2017 at 23:05:44 -0500, Stefan Monnier wrote:
> > I am proposing implementing islands as a solution to the problem, not as
> > a way of thinking about it.

> In order to design a solution, we need to think about the problem somehow.

That's all very well, but we could spend weeks (or months) talking about
the problem without getting anywhere.  With a proposed solution, we can
at least discuss whether it fixes the given problem, and if not, modify it
until it does.  You've already pointed out a sensible enhancement to it.
(Having a third new syntax class which can swap syntax tables without
nesting.)

> >> But the issue is that the syntax beginning in the above example should be
> >> point-min, not 1.

Incidentally, a parse state (like a voltage) is always a _difference_
between two points.  It is determined by (parse-partial-sexp start end
...).  It is an error in thinking to think that there is any a priori
syntax beginning in any buffer situation.

It is only the addition of a syntax cache which creates the notion of
this syntax beginning, but we must never forget that this syntax
beginning is a property of the cache, NOT of the narrowing.  Caches are
complicated things, but narrowing is simple and unambiguously defined.

> > Should it?

Here's that code fragment again, for reference:

    (save-restriction
      (narrow-to-region ...)
      (with-syntax-table ...
        (backward-sexp 1)))


> It's not really inherent in the code, but it's how it currently behaves
> (mostly), and in some cases that is what the author wants.  In other
> cases, the author wants something else.

That code appears to be from .../lisp/obsolete/complete.el, function
PC-lisp-complete-symbol.  If that's the case it's simply using narrowing
without any syntax cache, so there's no concept of a syntax beginning
whatsoever, there.  There would be no need to create an island in that
`narrow-to-region'.

> > When does it actually make a difference, apart from when point-min is
> > inside a string or a comment?  I don't think it makes a difference in
> > the (backward-sexp 1) case above.

> backward-sexp will call back_comment, so yes it makes a difference.
> And since speed may be important, it's also important to make sure we
> don't scan over the 100KB of text that comes before point-min.

:-)  There will be situations where things like backward-sexp will call
back_comment (which is why it is important that back_comment be fast)
but that code fragment isn't one of them.  And even if it did (which
will be rare), it is not doing it inside a tight loop.

> > That is correct.  The creation of an island will not be an expensive
> > action, unless it is in a tight loop - recording two current syntactic
> > elements, and calculating and setting two syntax-table text properties;
> > then restoring these later.

> But won't setting those islands flush the caches (e.g. comment-cache,
> syntax-ppss cache, ...)?

I think you've chosen a bad example for making that point.  The
syntax-ppss cache in the above code will need flushing anyway due to the
with-syntax-table.  The comment-cache cache might or might not need
flushing for that reason (depending on the differences between the two
syntax tables).

Creating an island will invalidate the syntax-ppss cache from the
beginning of the island, yes.  But few of the existing
`narrow-to-region's (there are ~750 in Emacs) will require to create
islands.  This is only needed where the newly created accessible portion
needs to be syntactically separated from what is outside of it.  In most
`narrow-to-region's (like that one in complete.el) this won't be the
case.

The only circumstances this would be concerning would be (i) creating
islands in a tight loop; and (ii) moving forward in the buffer in this
loop.  Are there any such circumstances in Emacs?

I honestly don't believe the flushing of caches is an important problem
in the proposed islands implementation.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Syntax ambiguities in narrowed buffers and multiple major modes: a proposed solution.

[Stefan Monnier]

>> In order to design a solution, we need to think about the problem somehow.
> That's all very well, but we could spend weeks (or months) talking about
> the problem without getting anywhere.

You think whichever way you want.  All I said was:

    [ I like to consider that strings and comments are also a form of
      "island", although we're probably better off supporting them in
      a special way like we do now.  ]

which I wrote mostly so as to give some background in how I think about
the problem so you can hopefully understand better my position.

You then started to argue that it's wrong to think this way.

>> >> But the issue is that the syntax beginning in the above example should be
>> >> point-min, not 1.
> Incidentally, a parse state (like a voltage) is always a _difference_
> between two points.

Here I disagree.  When you say "difference" it makes it sound like we
could take two differences and combine them.  That would be great: we
could cache every N chars the cumulated difference applied by those
N chars and then efficiently compute the state at any position by only
combining those pre-computed cumulated diffs.

But we can't do that, because the "parse state" is really a *state*.

> It is determined by (parse-partial-sexp start end ...).

Yes, the state depends on where we start parsing.  But there is
a privileged state which is the one rendered visible via font-lock, and
that's the one syntax-ppss intends to cache.

> It is an error in thinking to think that there is any a priori
> syntax beginning in any buffer situation.

I'm glad we agree about this.

> Here's that code fragment again, for reference:

>     (save-restriction
>       (narrow-to-region ...)
>       (with-syntax-table ...
>         (backward-sexp 1)))

>> It's not really inherent in the code, but it's how it currently behaves
>> (mostly), and in some cases that is what the author wants.  In other
>> cases, the author wants something else.

> That code appears to be from .../lisp/obsolete/complete.el, function
> PC-lisp-complete-symbol.

If so, it's a complete accident.  The fragment came straight out of
my imagination.  The situations I have in mind are more like in
perl-mode's syntax-propertize function where we need to find the
matching braces in regexp operations (where the matching rules are
slightly different from the ones in normal code) or in sgml-mode where
we jump from < to > and vice-versa using a specialized syntax-table, or
in sm-c-mode where I parse the C code within a CPP directive (itself
treated from the outside as a kind of comment).

> :-)  There will be situations where things like backward-sexp will call
> back_comment (which is why it is important that back_comment be fast)
> but that code fragment isn't one of them.  And even if it did (which
> will be rare), it is not doing it inside a tight loop.

I'm saying that the code fragment can be inside a tight loop (e.g. as
part of a backward lexer used for indentation purposes).

> I think you've chosen a bad example for making that point.  The
> syntax-ppss cache in the above code will need flushing anyway due to the
> with-syntax-table.  The comment-cache cache might or might not need
> flushing for that reason (depending on the differences between the two
> syntax tables).

Alan, I'm not trying to pimp syntax-ppss here or to put down
comment-cache, here.  I'm just pointing out a real existing problem
which I think the new design should take into account.  Indeed the
current syntax-ppss treatment is incorrect (and in sm-c-mode
I work around it by meddling in syntax-ppss's internals, which is
clearly a bad idea).

IOW, instead of trying to come up with ad-hoc ways to treat
narrow-to-region as something that places island markers, which are then
somehow removed by something else (presumably at the end of
save-restriction, tho that makes for ugly semantics, IOM) and then
additionally handle the with-syntax-table thingy, I think we should
design a new macro specifically for that kind of "temporarily work on
a region as if it was its own buffer, with its own syntax"
(i.e. combining the kind of effect usually obtained with
narrow-to-region+with-syntax-table+save-restriction).

Then we can implement it by adding island markers (and flush the cache)
if we want, and if that proves inefficient later on, we can change it to
use another implementation strategy.


        Stefan

Re: Syntax ambiguities in narrowed buffers and multiple major modes: a proposed solution.

[Stefan Monnier]

> If so, it's a complete accident.  The fragment came straight out of
> my imagination.  The situations I have in mind are more like in
> perl-mode's syntax-propertize function where we need to find the
> matching braces in regexp operations (where the matching rules are
> slightly different from the ones in normal code) or in sgml-mode where
> we jump from < to > and vice-versa using a specialized syntax-table, or
> in sm-c-mode where I parse the C code within a CPP directive (itself
> treated from the outside as a kind of comment).

FWIW, I looked at the sgml-mode parser and it looks pretty much like
that (a backward-sexp within a with-syntax-table within a loop), indeed,
except for one big difference: there's no narrowing (AFAICT the special
syntax table used to jump from > to the opening < just doesn't have
comments, luckily), and indeed it wouldn't be very clear where to start
the narrowing anyway (after all, the start is what we're looking for in
the first place).


        Stefan

Re: Syntax ambiguities in narrowed buffers and multiple major modes: a proposed solution.

[Dmitry Gutov]

On 28.02.2017 01:22, Stefan Monnier wrote:

> FWIW, I looked at the sgml-mode parser and it looks pretty much like
> that (a backward-sexp within a with-syntax-table within a loop)

ruby-syntax-propertize-percent-literal also has that.

Re: Syntax ambiguities in narrowed buffers and multiple major modes: a proposed solution.

[Alan Mackenzie]

Hello, Stefan.

On Mon, Feb 27, 2017 at 15:52:19 -0500, Stefan Monnier wrote:

[ .... ]

> All I said was:

>     [ I like to consider that strings and comments are also a form of
>       "island", although we're probably better off supporting them in
>       a special way like we do now.  ]

> which I wrote mostly so as to give some background in how I think about
> the problem so you can hopefully understand better my position.

> You then started to argue that it's wrong to think this way.

Sorry, that's not what I meant to say.  I simply wanted, in this thread,
to keep the meaning of "island" unambiguous.  That's the word, nothing
more.  This was to prevent the thread degenerating into confusion
between the concept "island" I have just defined, and the more general
uses and connotations attached to the word.

[ .... ]

> > It [a parse state] is determined by (parse-partial-sexp start end
> > ...).

> Yes, the state depends on where we start parsing.  But there is
> a privileged state which is the one rendered visible via font-lock, and
> that's the one syntax-ppss intends to cache.

That sounds like an intended resolution of the current ambiguity of the
starting position of syntax-ppss's cache.  Or am I reading too much into
the sentence?

[ .... ]

> > Here's that code fragment again, for reference:

> >     (save-restriction
> >       (narrow-to-region ...)
> >       (with-syntax-table ...
> >         (backward-sexp 1)))

[ .... ]

> > That code appears to be from .../lisp/obsolete/complete.el, function
> > PC-lisp-complete-symbol.

> If so, it's a complete accident.  The fragment came straight out of
> my imagination.  The situations I have in mind are more like in
> perl-mode's syntax-propertize function where we need to find the
> matching braces in regexp operations (where the matching rules are
> slightly different from the ones in normal code) or in sgml-mode where
> we jump from < to > and vice-versa using a specialized syntax-table, or
> in sm-c-mode where I parse the C code within a CPP directive (itself
> treated from the outside as a kind of comment).

OK.  But any time the current syntax-table is changed, the cache becomes
invalid.  For such operations, there really needs to be a means of
isolating the cache from the syntactic operations, and vice versa.

> > :-)  There will be situations where things like backward-sexp will call
> > back_comment (which is why it is important that back_comment be fast)
> > but that code fragment isn't one of them.  And even if it did (which
> > will be rare), it is not doing it inside a tight loop.

> I'm saying that the code fragment can be inside a tight loop (e.g. as
> part of a backward lexer used for indentation purposes).

OK, accepted.

> > I think you've chosen a bad example for making that point.  The
> > syntax-ppss cache in the above code will need flushing anyway due to the
> > with-syntax-table.  The comment-cache cache might or might not need
> > flushing for that reason (depending on the differences between the two
> > syntax tables).

> Alan, I'm not trying to pimp syntax-ppss here or to put down
> comment-cache, here.  I'm just pointing out a real existing problem
> which I think the new design should take into account.  Indeed the
> current syntax-ppss treatment is incorrect (and in sm-c-mode
> I work around it by meddling in syntax-ppss's internals, which is
> clearly a bad idea).

Also accepted, thanks.

> IOW, instead of trying to come up with ad-hoc ways to treat
> narrow-to-region as something that places island markers, which are then
> somehow removed by something else (presumably at the end of
> save-restriction, tho that makes for ugly semantics, IOM) and then
> additionally handle the with-syntax-table thingy, I think we should
> design a new macro specifically for that kind of "temporarily work on
> a region as if it was its own buffer, with its own syntax"
> (i.e. combining the kind of effect usually obtained with
> narrow-to-region+with-syntax-table+save-restriction).

> Then we can implement it by adding island markers (and flush the cache)
> if we want, and if that proves inefficient later on, we can change it to
> use another implementation strategy.

The prime motivator for islands is syntactically to mark the various
regions of a multi-major-mode buffer which have their own syntax tables,
and to do this in a manner which doesn't impose any restrictions or
complications on programs' or users' use of narrowing.  It is evident
that that same mechanism could be useful for marking a "neutral point"
for syntactic analysis.

Up to now I've concentrated on the pure mechanism without much
consideration of the interface with the rest of Lisp, though that is
clearly important, too.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Syntax ambiguities in narrowed buffers and multiple major modes: a proposed solution.

[Stefan Monnier]

>> Yes, the state depends on where we start parsing.  But there is
>> a privileged state which is the one rendered visible via font-lock, and
>> that's the one syntax-ppss intends to cache.
> That sounds like an intended resolution of the current ambiguity of the
> starting position of syntax-ppss's cache.  Or am I reading too much into
> the sentence?

I think you're reading too much into it ;-)
E.g. in order for font-lock to give the right result, we sometimes need
to run it (and syntax-ppss) on an "island".

> OK.  But any time the current syntax-table is changed, the cache becomes
> invalid.  For such operations, there really needs to be a means of
> isolating the cache from the syntactic operations, and vice versa.

That's right.  But I think it's important to be able to *temporarily*
invalidate the cache (so as soon as you leave the with-syntax-table,
the old cache is reinstated).


        Stefan

Re: Syntax ambiguities in narrowed buffers and multiple major modes: a proposed solution.

[Alan Mackenzie]

Hello, Stefan.

On Tue, Feb 28, 2017 at 14:09:54 -0500, Stefan Monnier wrote:
> >> Yes, the state depends on where we start parsing.  But there is
> >> a privileged state which is the one rendered visible via font-lock, and
> >> that's the one syntax-ppss intends to cache.

> > That sounds like an intended resolution of the current ambiguity of the
> > starting position of syntax-ppss's cache.  Or am I reading too much into
> > the sentence?

> I think you're reading too much into it ;-)
> E.g. in order for font-lock to give the right result, we sometimes need
> to run it (and syntax-ppss) on an "island".

Just to emphasize, in my "island" scheme, the syntactic state by
starting "at the island" would be the same as starting at BOB.  Either
would work.  This should simplify the syntactic handling.

> > OK.  But any time the current syntax-table is changed, the cache becomes
> > invalid.  For such operations, there really needs to be a means of
> > isolating the cache from the syntactic operations, and vice versa.

> That's right.  But I think it's important to be able to *temporarily*
> invalidate the cache (so as soon as you leave the with-syntax-table,
> the old cache is reinstated).

I agree with you here.  I'll be thinking about it.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Syntax ambiguities in narrowed buffers and multiple major modes: a proposed solution.

[Alan Mackenzie]

Hello, Stefan.

On Tue, Feb 28, 2017 at 20:27:20 +0000, Alan Mackenzie wrote:
> On Tue, Feb 28, 2017 at 14:09:54 -0500, Stefan Monnier wrote:

[ .... ]

> > > OK.  But any time the current syntax-table is changed, the cache becomes
> > > invalid.  For such operations, there really needs to be a means of
> > > isolating the cache from the syntactic operations, and vice versa.

> > That's right.  But I think it's important to be able to *temporarily*
> > invalidate the cache (so as soon as you leave the with-syntax-table,
> > the old cache is reinstated).

> I agree with you here.  I'll be thinking about it.

How about the following proposal: we introduce the notion of
@dfn{indirect text properties}.  We give the symbol 'syntax-table a
property 'indirect-text-property (or possibly some better name) whose
value is a symbol 'foo (not necessarily interned).

Any access, read or write, to a syntax-table text property would find
the 'indirect-text-property property, and read or write the 'foo text
property instead.  This checking of 'indirect-text-propery would,
naturally, be done in the C code for text properties, not in Lisp.

In our use case, when we need *temporarily* to invalidate the cache, we
would simply change the 'indirect-text-property property on
'syntax-table to 'foo-temp.  When we've finished using it, we swap back,
possibly with garbage collection tricks of some sort, to make sure the
'foo-temp text properties were cleared from the buffer.

What do you think?

> >         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Syntax ambiguities in narrowed buffers and multiple major modes: a proposed solution.

[Filipp Gunbin]

Hi Alan, 

How your proposal would work when the island starter consists of several
characters?  Like an XML comment, or maybe Java code embedded into JSP.

On which characters the island syntax class will be placed?

Sorry if I'm missing something obvious.

Filipp.

Re: Syntax ambiguities in narrowed buffers and multiple major modes: a proposed solution.

[Tom Tromey]

>>>>> "Alan" == Alan Mackenzie <acm@muc.de> writes:

Alan>   o - Each time parse-partial-sexp encounters an island open, it pushes
Alan>     the current state, including the syntax table, onto this stack,
Alan>     reinitialises the state, and optionally sets the syntax table, for
Alan>     the island just being opened.  The island begins at the buffer
Alan>     position _after_ the one bearing the island open syntax.

This approach means that an island can't start at the beginning of a
buffer.  I don't have an example offhand where this matters, but it
seems like an arbitrary restriction, and furthermore one not imposed by
the current approach of having syntax table property on the affected
characters.  Maybe an island could be delimited in this same way, that
is, an island would be a contiguous group of characters that have a
property ('island, or whatever) with the same value.

Alan>   o - A new function would be needed to get the "base" syntax of a
Alan>     position, the syntax it would have if it weren't for any island
Alan>     open/close syntax table properties on it.

Would you mind giving an example of what this would be used for?

Tom

Re: Syntax ambiguities in narrowed buffers and multiple major modes: a proposed solution.

[Alan Mackenzie]

Hello, Tom.

I know it's almost three weeks ago, but....

On Sat, Mar 04, 2017 at 12:41:23 -0700, Tom Tromey wrote:
> >>>>> "Alan" == Alan Mackenzie <acm@muc.de> writes:

> Alan>   o - Each time parse-partial-sexp encounters an island open, it pushes
> Alan>     the current state, including the syntax table, onto this stack,
> Alan>     reinitialises the state, and optionally sets the syntax table, for
> Alan>     the island just being opened.  The island begins at the buffer
> Alan>     position _after_ the one bearing the island open syntax.

> This approach means that an island can't start at the beginning of a
> buffer.  I don't have an example offhand where this matters, but it
> seems like an arbitrary restriction, and furthermore one not imposed by
> the current approach of having syntax table property on the affected
> characters.  Maybe an island could be delimited in this same way, that
> is, an island would be a contiguous group of characters that have a
> property ('island, or whatever) with the same value.

I've been thinking about this.  One problem would be temporarily putting
an island in the middle of another one.  When the time came to remove
that enclosed island, the 'island property would need to be restored,
somehow.  It might get messy if a "temporary" island T crosses the
boundary between "permanent" islands, A and B, like this:

    ..........AAAAAAAAAAAAAAAABBBBBBBBBBBBB.............
                         TTTTTTTTTT

.  It's the difference between marking end points, and covering an
interval.  In the first, the markers take up space; in the second what
gets covered sometimes needs uncovering.

I think this could work, but needs more thought.

> Alan>   o - A new function would be needed to get the "base" syntax of a
> Alan>     position, the syntax it would have if it weren't for any island
> Alan>     open/close syntax table properties on it.

> Would you mind giving an example of what this would be used for?

It was something Dmitry said a few weeks back, which I can't quite
remember.  It was something to do with an island ending at an EOL, which
would have the EOL's syntax superseded by an island close, thus damaging
other code which needs to see the EOL's syntax rather than the island
close syntax.

Maybe a better way to implement islands would be to have new text
properties to mark their boundaries rather than extending 'syntax-table.
The syntax code would recognise and handle the new 'island-open
property, yet this wouldn't obliterate the "base" syntax of a position.

This way, it might be possible to have the island beginning at the
position the 'island-open property is on, and ending at the
'island-close property.  That would allow islands as small as two
character positions.  It might even allow us islands as small as a
single character position, if we can put 'island-open and 'island-close
on the same character position.

Or something like that.

> Tom

-- 
Alan Mackenzie (Nuremberg, Germany).