Better parse-partial-sexp; multiple major modes (was: Idea for syntax-ppss)

Better parse-partial-sexp; multiple major modes (was: Idea for syntax-ppss)

[Daniel Colascione]

On Jul 27, 2008, at 10:50 AM, Alan Mackenzie wrote:
> What I think really needs doing is to make this function  
> bulletproof: It
> should work on narrowed buffers, it should give reliable elements 2  
> and
> 6, its cache should be cleared when functions like `modify-syntax- 
> entry'
> are called or parse-sexp-lookup-properties is changed, and the cache
> should be bound to nil on `with-syntax-table'.  I actually think it
> could be useful to maintain several parallel caches, each for a
> different syntax-table (or an equivalence class of syntax tables).   
> And
> so on.  Basically, I would like `(syntax-ppss)' to tell me with 100%
> reliability, no ifs, no buts, whether I am at top-level, in a comment,
> or in a string.

Surely, such a creature would have to live on the C side of things, if  
only for practical reasons. (With the proliferation of with-this and  
inhibit-that options available to Lisp, I don't see how one can easily  
and robustly catch all buffer modification. Not to mention that no  
matter which of before-change-functions and after-change-functions you  
used, you could still race against other functions using the same  
facility.)

If this perfectly caching parse-partial-sexp lives in C anyway, why  
not just call it parse-partial-sexp? Optimize parse-partial-sexp for  
the case of start being 1 or (point-min). syntax-ppss becomes a simple  
wrapper. Not only would it be possible to robustly catch all buffer  
and context modification, but by optimizing the existing function, all  
existing users would automatically win. I'd offer to write a patch,  
but I don't know the core well enough to know how to "easily and  
robustly catch all buffer modification".

>
> Also, Lennart is asking for it to work nicely with multiple major  
> modes.
> Surely this would be a Good Thing.  Files containing several major  
> modes
> are commonplace (awk or sed embedded within a shell script, html
> embedded within php, ....).

After several attempts at using and understanding multiple major mode  
facilities, I'm convinced the only way forward is core support for the  
concept. Lennart's done a fine job with what's in Emacs currently. But  
anything involving multiple major modes today is a quivering mound of  
hacks. All the work Lennart's had to do to get modes playing nice with  
each other is a testament to that.

Maybe a core solution could be something like this: in a given buffer,  
each character has a chunk-name character property. You'd buffer- 
locally map chunk names to major modes. For each chunk name, create a  
buffer containing just the text assigned to that chunk. Make the major- 
mode the major mode for the chunk buffer, and let that major-mode  
handle fontification, keybindings, and so on. In the main buffer,  
assemble the various bits from the chunk-buffers and allow the user to  
navigate the combined buffer normally.

Keybindings with point at a particular character would just be the  
keybindings present in that character's chunk-buffer. If you need  
special keybindings common across all chunk buffers, just bind the key  
in all the chunk buffers. If a given chunk needs placeholder text to  
represent text of some other chunk, it should be possible add it to  
that chunk buffer without affecting any of the others.

Anyway, this scheme is:

1) Robust - no messing around with variables, no tweaking fontification
2) Backwards compatible - a major-mode doesn't need to know it's being  
used this way
3) Versatile - you can compose arbitrary modes this way, even  
recursively
4) Conceptually simple (I hope)

Any thoughts?

Better parse-partial-sexp; multiple major modes (was: Idea for syntax-ppss)

[Daniel Colascione]

On Jul 27, 2008, at 10:50 AM, Alan Mackenzie wrote:
> What I think really needs doing is to make this function  
> bulletproof: It
> should work on narrowed buffers, it should give reliable elements 2  
> and
> 6, its cache should be cleared when functions like `modify-syntax- 
> entry'
> are called or parse-sexp-lookup-properties is changed, and the cache
> should be bound to nil on `with-syntax-table'.  I actually think it
> could be useful to maintain several parallel caches, each for a
> different syntax-table (or an equivalence class of syntax tables).   
> And
> so on.  Basically, I would like `(syntax-ppss)' to tell me with 100%
> reliability, no ifs, no buts, whether I am at top-level, in a comment,
> or in a string.

Such a thing would have to live on the C side of things, right? (With  
the proliferation of with-this and inhibit-that options available to  
Lisp, I don't see how one can easily and robustly catch all buffer  
modification. Not to mention that no matter which of before-change- 
functions and after-change-functions you used, you could still race  
against other functions using the same facility.)

If this perfectly caching parse-partial-sexp lives in C anyway, why  
not just call it parse-partial-sexp? Optimize parse-partial-sexp for  
the case of start being 1 or (point-min). syntax-ppss becomes a simple  
wrapper. Not only would it be possible to robustly catch all buffer  
and context modification, but by optimizing the existing function, all  
existing users would automatically win. I'd offer to write a patch,  
but I don't know the core well enough to know how to "easily and  
robustly catch all buffer modification".

> Also, Lennart is asking for it to work nicely with multiple major  
> modes.
> Surely this would be a Good Thing.  Files containing several major  
> modes
> are commonplace (awk or sed embedded within a shell script, html
> embedded within php, ....).

After several attempts at using and understanding multiple major mode  
facilities, I'm convinced the only way forward is core support for the  
concept. Lennart's done a fine job with what's in Emacs currently. But  
anything involving multiple major modes today is a quivering mound of  
hacks. All the work Lennart's had to do to get modes playing nice with  
each other is a testament to that.

Maybe a core solution could be something like this: in a given buffer,  
each character has a chunk-name character property. You'd buffer- 
locally map chunk names to major modes. For each chunk name, create a  
buffer containing just the text assigned to that chunk. Make the major- 
mode the major mode for the chunk buffer, and let that major-mode  
handle fontification, keybindings, and so on. In the main buffer,  
assemble the various bits from the chunk-buffers and allow the user to  
navigate the combined buffer normally.

Keybindings with point at a particular character would just be the  
keybindings present in that character's chunk-buffer. If you need  
special keybindings common across all chunk buffers, just bind the key  
in all the chunk buffers. If a given chunk needs placeholder text to  
represent text of some other chunk, it should be possible add it to  
that chunk buffer without affecting any of the others.

Anyway, this scheme is:

1) Robust - no messing around with variables, no tweaking fontification
2) Backwards compatible - a major-mode doesn't need to know it's being  
used this way
3) Versatile - you can compose arbitrary modes this way, even  
recursively
4) Conceptually simple (I hope)

Any thoughts?

Re: Better parse-partial-sexp; multiple major modes

[Stefan Monnier]

> Such a thing would have to live on the C side of things, right? (With the
> proliferation of with-this and inhibit-that options available to Lisp,
> I don't see how one can easily and robustly catch all buffer
> modification.  Not to mention that no matter which of before-change-
> functions and after-change-functions you used, you could still race against
> other functions using the same facility.)

In theory, that might be true, but it's not been that big of a deal
for now.  And of course moving it to C would help some but it won't
solve all those problems magically.

> Anyway, this scheme is:

> 1) Robust - no messing around with variables, no tweaking fontification
> 2) Backwards compatible - a major-mode doesn't need to know it's being used
> this way
> 3) Versatile - you can compose arbitrary modes this way, even recursively
> 4) Conceptually simple (I hope)

I don't think it's possible to get 1 and 2 together, really.


        Stefan

Re: Better parse-partial-sexp; multiple major modes (was: Idea for syntax-ppss)

[Richard M. Stallman]

    Maybe a core solution could be something like this: in a given buffer,  
    each character has a chunk-name character property. You'd buffer- 
    locally map chunk names to major modes. For each chunk name, create a  
    buffer containing just the text assigned to that chunk.

That is elegant but very expensive.  Making so many buffers display as
one buffer, or keeping them in sync, would mean major changes at a low
level.  Even coming up with a spec for how insertion and deletion should
work in such a scheme is hard.

We are much better off with MuMaMo.