Re: Update 1 on Bytecode Offset tracking

[Zach Shaftel <zshaftel@gmail.com>]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Great! I just followed up on my copyright assignment as I still haven't
>> finished that process. I don't know whether this could be exempt or if
>> Rocky's assignment is sufficient, but hopefully I will hear back from
>> copyright-clerk soon.
>
> While waiting for the paperwork to go through, you can prepare the patch
> and we can start discussing it.

Sure, does that just mean the 'git format-patch -1' emailed to
bug-gnu-emacs@gnu.org, as mentioned in CONTRIBUTE? If that's the gist of
it then I can do that shortly.

>> That seems to be the case.  I'll keep looking to see if there's any low
>> hanging fruit in terms of splitting up the funcall logic without slowing
>> things down.  More testing is necessary, but if a moderate chunk of
>> duplicated code is acceptable then there may not be as much work needed
>> on that branch as I had thought.
>
> It's a tradeoff, so it's hard to say what is acceptable without seeing
> the actual path along with the corresponding measurements of the
> performance impact.
>
>> Rough tests indicate it's about three times slower.
>
> Wow!  That's a lot less than I expected.  That makes it quite usable.

I was able to speed that function up to the point that it's about the
same as one using `read`. Those functions are doing a whole lot of IO
(reading and writing hundreds of files) so it's not really a fair
comparison. I've done more tests with functions that just read a whole
buffer, collecting what they read into a list. In a 9600 line file with
just over 500 sexps, the `read` version took about ~.02-.04 seconds
(according to `benchmark-run-compiled`), and the `source-map-read`
version took ~.08 seconds when it didn't GC, but unlike with `read` it
did cause a GC 10-20% of the time.

> This said, we'll probably still want to merge the feature into the
> C code simply to avoid the duplication (I expect that the Edebug reader
> has never been 100% faithful and that it has probably diverged over
> time).

I already had a couple minor issues with edebug's reader, though they
were easily remedied. It's pretty hard to hook into it since it relies
heavily on dynamic binding. So I agree edebug is not the ideal candidate
for a final product.

>> Removing the string did improve performance, but not by as much as I
>> expected. The function that constructs the tree of "children" may be
>> slower than it needs to be, so I'll look into improving that. It may not
>> be necessary to create the children for vectors since they're constants
>> (outside of backquote, at least).
>
> I think vectors are rare enough that the performance benefit of special
> casing them is not worth the downside of losing source-location info
> when it'd be beneficial.
>
>> Yes, and it won't be easy to maintain the read locations across
>> macroexpansion, byte-opt and cconv.
>
> For byte-opt and cconv it's mostly a question of labour.
>
> For macros, OTOH, it's really fundamentally hard (or impossible, in
> general).

Helmut Eller mentioned before that most macros do use at least some of
the original code in their expansion. A crude test with a pair of
hash-tables indicates 120/594 of the conses in the definition of
`cl-typep` remained after `macroexpand-all`ing. That's not much, but
considering how macro-laden the function is, that doesn't seem totally
impossible to work with.

> We could/should introduce some new way to define macros which
> knows about "source code annotated with locations".

I've wondered about this too but don't know what the right approach
would be. I doubt anyone would want to use something like
macro-cons/list/append etc. functions, and somehow modifying the
behavior of the real list operations during macroexpansion seems
impractical. I'm sure someone can come up with a good approach to this.

> There's a lot of work on Scheme macros we could leverage for that.

Interesting, so far I've had some difficulty finding documentation about
how other Lisps track source locations. If you know where I can find
some more details about that I'd love to take a look.

-Zach