Re: Update 1 on Bytecode Offset tracking

[Zach Shaftel <zshaftel@gmail.com>]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Sure, I'll do some more exhaustive testing. So far though, the results
>> aren't great, the biggest issue being memory usage. The
>> `source-map-read` can GC over 5 times more often than `read`.
>
> Sounds fine for a prototype.
>
>> I think it's more sensible to accept that a real implementation will
>> have to be in C and this reader will just remain a prototype.
>
> Indeed.
>
>> Aha, I had never even considered hygienic macros in Elisp (nor had I
>> recognized how trivial it is to track their source-code).  That would be
>> an amazing development for Emacs Lisp, but is certainly a huge
>> undertaking, not something I could fit into the GSoC timeline.
>
> No, I'm just discussing what the longer-run might look like.
>
>> I know that it has been done in Common Lisp (by Pascal Costanza), but
>> I believe that implementation serves the sole purpose of capture
>> avoidance and doesn't abstract syntax. For Emacs I assume this would
>> have to be done in C, but I do wonder if an Elisp implementation would
>> be possible.
>
> I haven't thought very much about it, but I can't see any reason why it
> would need to be done in C, no (tho I wouldn't be surprised if it could
> benefit from a bit of help from the C side, of course).

This is my gut feeling as well, but in the few discussions I've seen
about implementing hygiene in an unhygienic macro system people suggest
that it's nigh impossible without rewriting core parts of the language.
I haven't seen a convincing defense of that argument in any of those
discussions though.

>> Doing the similar thing in Elisp -- relegating source location tracking
>> to code using only a specialized kind of macro, hygienic or otherwise --
>> would of course be a major loss, since it would take years for that new
>> paradigm to become commonplace.
>
> Indeed, we'll need some fallback heuristic for all the existing
> `defmacro`s.
>
> Part of the issue is "tracking source location" but another important
> part is to take the annotated source code and "de-annotate" it
> (recursively) to pass it to the macro, since the macro expects
> a raw sexp.
>
> That's why we've been thinking about annotated representations
> which are "transparent" (i.e. can be used as if they weren't annotated).
> Either using "fat cons-cells" or using "fat symbols" or storing the
> annotations in an eq-hash-table.

A hash-table seems like the most straightforward approach, which I'm
working on now. Doing something like in the scratch/accurate-warning-pos
branch adds a whole lot of complexity to both C and Lisp, and toggling
byte-compilation versions of subrs feels clunky to me (though that's
obviously just a prototype). A hash-table of conses will hopefully be
enough, with or without the `source-map` stuff.

> Another way to attack the problem is to rely on the Edebug spec: you can
> refrain from de-annotating all the parts marked as `form` or `body` (as
> long as the annotations themselves look sufficiently like normal code,
> at least).

Interesting, that's not something I had thought about. I suspect flawed
edebug specs are common enough that this can't be relied upon.


-Zach