Re: Adding refactoring capabilities to Emacs

[Lynn Winebarger <owinebar@gmail.com>]

On Thu, Sep 7, 2023 at 1:08 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> >> Going back from there to the actual source code can be a fair bit more
> >> tricky,
> > Exactly.
> >> but Alan's symbols-with-positions should provide just what we need.
> > Great! but how?
> > Some years ago I presented a full source-tracking reader for Common Lisp
> > with just Lisp to a conference ;-) Needs a programmable reader though :-(
> > As you may remember, I then paired it with a full code walker to create
> > a stepper.
>
> We do have a "full source-tracking reader" writen in ELisp inside `edebug.el`.
> It's not 100% faithful to `lread.c`, but it seems to work well enough
> that we haven't heard too many bug reports about it over the years.
>
> Still, can't be used with `cconv` because the the "sexp with
> source-tracking info" don't have the shape expected by macros.

I have a package "rewriting-pcase" which contains the code I use to
textually rewrite some very specific, small sexps in packages so the
data files and subdirectories do not have to be located in a
particular location relative to the source library.  The text is
rewritten to avoid dramatic changes in the formatting of the original
code, but it only works because the sexp patterns are not complicated,
and the replacement sexps are atoms.

To expand the functionality to be true to its name, I was planning to
augment the wrap the reader with an elisp function to create
"syntax-objects" (as they are called in Scheme macro expanders) that
augment the data produced by the reader with various metadata,
including things like source file, line number, position, as well as
lexical information for expansion hygiene.  For rewriting-pcase, I'd
also track comments that should "attach" to the syntactic construct so
comments would move around with the code they describe when that code
is substituted while rewriting.

"symbol-with-position" appears to just be a handle on a symbol with a
position.  Would it be acceptable to change this to a more generic
syntax-object, then factor the reader into an enhanced reader that
produces syntax-objects, and provide the traditional read function as
a wrapper that discards the syntax?  Is the existence of position
information relied on in many places in the code?  I would think those
places could use the syntax-object-producing version of the reader.

My current approach is to recursively use the reader to deduce the
required syntactic information, but in addition to being expensive,
it's also unreliable in many cases that seem like they would be corner
cases but do occur in the wild, where the same datum can be
represented in multiple ways and are tricky to navigate with the usual
syntax-traversing functions used in working with elisp buffers.

Lynn