Re: Overlay mechanic improvements

[Richard Stallman <rms@gnu.org>]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    The problem here is that tokens are not just lexemes(token text). To
    be general and granular enough one also has to remember dependencies
    between tokens. So, a token is: ["lexeme", lookback, lookahead,
    position].

Ok, but why is it a "problem"?  It seems easy enough to construct a
list (LEXEME LOOKBACK LOOKAHEAD) and put it as the value of the
`token' property for all the characters in the token.

There is no need to include POSITION in this list, since you can
find the start of the token by calling `previous-char-property-change'.

    >      All I need
    >     is an ability to save position pairs, the positions should survive text
    >     insertion/deletion
    >
    > I see multiple meanings for that; could you clarify?

    I mean something like relative positioning of markers/overlays:
    changes in unrelated buffer parts should change positions accordingly.

Text properties do that.

    Yes, properties do survive copying, but token context might change and
    the whole thing will have to be reparsed anyway.

If it needs to be reparsed, that's no problem; it is easy to delete
the `token' property from a region.  In fact, we can specify automatic
deletion of a certain text property from text copied into the buffer.

But there are many cases where it does not need to be reparsed.  If
you can detect such cases, and reparse only near the boundaries, it
would be a nice optimization for cases like killing a whole function
and yanking it between two other functions.

You could use `yank-handler' to hook this in.  It would check whether
the existing parsing of the text being yanked fits the new context; if
so, it would preserve the `token' property; if not, it would erase the
`token' property of the yanked text and cause it all to be reparsed.

Another idea: even if the beginning of the yanked text has to be
interpreted differently in the new context, maybe tokens later on
in the yanked text will still be right in the new context.  If you can
detect when this happens, you might be able to skip reparsing of
some of the yanked text.  That could be a substantial speedup.

    Citing the "Special
    properties" documentation page for text properties: "you can't predict
    how many times the function will be called". One can imagine a
    workaround for this, but it would be just too cumbersome.

If you put a timestamp into the token data structure, it will be
easy to detect duplicate calls, and make them do nothing.

I think it would be less work to change text properties to give you
whatever interface features are convenient, than to redesign how
overlays are stored.

    After all overlays do have to be fixed anyway.

Why so?  Aside from not being suitable for this use,
what is wrong with overlays that needs to be fixed?

The feature you want to implement is very useful, and we should
provide the low-level support needed to make it clean and fast.  But
if that is easy to do with text properties, let's do it with text
properties.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.