Syntax ambiguities in narrowed buffers and multiple major modes: a proposed solution.

[Alan Mackenzie <acm@muc.de>]

Hello, Emacs.

This post proposes a method of coding up syntactic "islands" as a
solution to various syntactic problems in the current Emacs.  It aims to
address the various problems raised in the thread Subject: Bug #25608
and the comment-cache branch.

(i) Problems:
  o - Ambiguity involved with narrowed regions - sometimes programs and
    users wish to see syntactic entities (e.g. strings, comments) as
    though point-min were a syntactically neutral position - other times
    they want the syntax to be relative to the beginning of the buffer.
  o - Multiple major modes - each submode region should start
    syntactically at a syntactically neutral position, regardless of the
    syntactic context of the enclosing supermode.  This region should be
    able to have its own syntax table.

(ii) Proposed solution - islands.
  o - By default, in a narrowed region, the syntactic context of a
    position will be that relative to BOB.
  o - There will be two new syntax classes introduced for use in syntax
    table text properties: "island open" and "island close".  Together,
    these enclose an "island", a region of the buffer syntactically
    disjoint from the text outside of the region.
  o - An island open syntax class will have, in place of the matching
    character argument (used by open/close parenthesis classes), a list
    argument containing:
    * - the (optional) syntax table for the island;
    * - the "previous syntax" of the position, which has been overwritten
      by the island open class syntax-table text property.
    * - Possible further items.
  o - An island close syntax class will have, in place of the
    matching character argument, a list containing:
    * - The "previous syntax" of the position.
    * - Possible further items.

(iii) Effect on parse-partial-sexp/syntax-ppss.
  o - The 10-element parser state would become a stack of parser states.
  o - Each time parse-partial-sexp encounters an island open, it pushes
    the current state, including the syntax table, onto this stack,
    reinitialises the state, and optionally sets the syntax table, for
    the island just being opened.  The island begins at the buffer
    position _after_ the one bearing the island open syntax.
  o - Each time parse-partial-sexp encounters an island close, it pops
    the suspended state, including the syntax table, off the stack of
    states.  (If there is no stacked state, the function stops at this
    point, somewhat analagously to reaching the end of buffer.)
  o - parse-partial-sexp would be amended to allow the caller to direct
    the parsing to stop at the end (and possibly beginning) of an island,
    much as it currently does for strings and comments.

(iv) Modification of certain commands and functions.
  o - narrow-to-region will be given an optional argument which, if set,
    directs Emacs to make the new region an island.  Thus, C-u C-x n n
    would enable a user to narrow to a "comment within a string" and edit
    it as though it were a comment.
  o - A new function would be needed to get the "base" syntax of a
    position, the syntax it would have if it weren't for any island
    open/close syntax table properties on it.
  o - forward-comment will need to cope, one way or another, with the
    possibility of an island inside a comment.  Several other scanning
    commands will likewise need amendment.

(v) Advantages of this proposal.
  o - The ambiguity in narrowed regions would be resolved without loss of
    functionality.
  o - The fact that the syntactic context is always global in a buffer
    facilitates the building and manipulation of syntax caches (of
    whatever nature).
  o - Multiple major mode modes could be implemented without abusing the
    narrowing mechanism.

-- 
Alan Mackenzie (Nuremberg, Germany).