Circular records: how do I best handle them? (The new correct warning position branch now bootstraps in native compilation!)

Circular records: how do I best handle them? (The new correct warning position branch now bootstraps in native compilation!)

[Alan Mackenzie]

Hello, Emacs.

Firstly, the scratch/correct-warning-pos git branch (which is to deliver
the correct source code position in byte compilation warning messages)
bootstrapped with native compilation for the first time yesterday
evening.  Basically, it is now working.  :-)

However, several files.el failed to compile due to infinite recursion.
A bit of gdb'ing in Ffuncall showed that whilst compiling ffap.el (at
least), there were circular record structures (where @dfn{records} are a
pseudovector type very like normal vectors, but with their zeroth
element supposedly representing their type with a symbol).

The recursion was in the (new) function `byte-compile-strip-s-p-1'
(where "strip-s-p" stands for "strip-symbol-positions").  This function
recursively descends list and vector structures, stripping positions
from symbols-with-position it finds.

However, I seem to have circularity in a record structure, where two
records seem to point to eachother.  I suspect that it is the zeroth
elements of these records (which are meant to be symbols) that are
pointing at eachother.

Would somebody (?Stefan M, perhaps) please suggest to me how I might
efficiently cope with these circular structures.  Do I need to maintain
a list of already encountered Lisp Objects, somehow, and check this list
before recursing into the function?  It would be nice if the new
mechanism were reasonably efficient (but I don't know how important that
is).

Thanks!

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Circular records: how do I best handle them? (The new correct warning position branch now bootstraps in native compilation!)

[Stefan Monnier]

> The recursion was in the (new) function `byte-compile-strip-s-p-1'
> (where "strip-s-p" stands for "strip-symbol-positions").  This function
> recursively descends list and vector structures, stripping positions
> from symbols-with-position it finds.
>
> However, I seem to have circularity in a record structure, where two
> records seem to point to eachother.  I suspect that it is the zeroth
> elements of these records (which are meant to be symbols) that are
> pointing at eachother.

Hmm... circularity is quite normal in data structures, yes.
But presumably this is only applied to source code where circularity is
very rare.  Could it be that you end up recursing in elements which
actually aren't part of the source code (and hence can't have
symbols-with-positions)?

Also, I see in `byte-compile-strip-s-p-1` that you only look inside conses
and vectors.  So I'm not sure what makes you say the recursion was in
records since records are similar to vectors but aren't `vectorp` so
AFAICT your code won't recurse into them.

Also that means your code won't handle the case where the source code
includes literal hash-tables, literal records, literal char-tables,
literal strings-with-properties, ...

These are quite rare and maybe it's OK to disallow them in source code,
but maybe a more robust approach would be to make sure your lread.c code
doesn't generate symbols-with-positions within anything else than conses
and vectors?
[ Tho it wouldn't prevent a macro from expanding into a literal hash-table
  from source that only has conses&vectors :-(  ]

> Would somebody (?Stefan M, perhaps) please suggest to me how I might
> efficiently cope with these circular structures.  Do I need to maintain
> a list of already encountered Lisp Objects, somehow, and check this list
> before recursing into the function?

That's what we do elsewhere, yes, except that history taught us that
a hash-table is a better choice to avoid scalability problems.
Tho in your case you'd only need to keep the stack of objects inside of
which you're currently recursing, so maybe a list is good enough.

BTW, instead of doing it by hand, you can call `print--preprocess` which
will do the recurse-and-fill-hash-table for you.  You can see how it's
used in `cl-print.el`.


        Stefan

Re: Circular records: how do I best handle them? (The new correct warning position branch now bootstraps in native compilation!)

[Alan Mackenzie]

Hello, Stefan.

On Fri, Dec 24, 2021 at 08:56:56 -0500, Stefan Monnier wrote:
> > The recursion was in the (new) function `byte-compile-strip-s-p-1'
> > (where "strip-s-p" stands for "strip-symbol-positions").  This function
> > recursively descends list and vector structures, stripping positions
> > from symbols-with-position it finds.

> > However, I seem to have circularity in a record structure, where two
> > records seem to point to eachother.  I suspect that it is the zeroth
> > elements of these records (which are meant to be symbols) that are
> > pointing at eachother.

> Hmm... circularity is quite normal in data structures, yes.
> But presumably this is only applied to source code where circularity is
> very rare.  Could it be that you end up recursing in elements which
> actually aren't part of the source code (and hence can't have
> symbols-with-positions)?

I honestly don't know at the moment.

> Also, I see in `byte-compile-strip-s-p-1` that you only look inside conses
> and vectors.  So I'm not sure what makes you say the recursion was in
> records since records are similar to vectors but aren't `vectorp` so
> AFAICT your code won't recurse into them.

byte-compile-strip-s-p-1 has been enhanced to handle records, too,
though I haven't committed that bit yet (along with quite a lot of other
amendments, too).

> Also that means your code won't handle the case where the source code
> includes literal hash-tables, literal records, literal char-tables,
> literal strings-with-properties, ...

The positions get stripped off hash-table keys in puthash.  I'm unsure
about the others, just offhand.

Put it this way, make bootstrap is currently working, although a bit
delicate.  My preliminary timings on a benchmark are as fast as
expected, so it's looking good.

> These are quite rare and maybe it's OK to disallow them in source code,
> but maybe a more robust approach would be to make sure your lread.c code
> doesn't generate symbols-with-positions within anything else than conses
> and vectors?
> [ Tho it wouldn't prevent a macro from expanding into a literal hash-table
>   from source that only has conses&vectors :-(  ]

> > Would somebody (?Stefan M, perhaps) please suggest to me how I might
> > efficiently cope with these circular structures.  Do I need to maintain
> > a list of already encountered Lisp Objects, somehow, and check this list
> > before recursing into the function?

> That's what we do elsewhere, yes, except that history taught us that
> a hash-table is a better choice to avoid scalability problems.
> Tho in your case you'd only need to keep the stack of objects inside of
> which you're currently recursing, so maybe a list is good enough.

I've tried the list approach (using memq to check for an already
processed cons/vector/record.  It fell flat on its face with
lisp/leim/ja-dic/ja-dec.el, which has a list with over 60,000 strings in
it.  I Ctrl-C'd out of this after five minutes, and it took me a while
to establish I didn't have an infinite loop; not quite.

So I disabled the checking for circularity in conses, leaving it in for
vectors and records.  That's what I meant when I said "a bit delicate"
above.  There's nothing to stop somebody building a circular list in a
source file.  Maybe the way to handle this would be to allow it to hit
max-lisp-eval-depth, catch the error, then turn on the circularity
detector and try again.  Circular lists in source code are surely rare.
Large circular lists must be rarer still.

> BTW, instead of doing it by hand, you can call `print--preprocess` which
> will do the recurse-and-fill-hash-table for you.  You can see how it's
> used in `cl-print.el`.

Thanks, I'll have a closer look at this, probably in a few days time.

Have a good tomorrow!

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Circular records: how do I best handle them? (The new correct warning position branch now bootstraps in native compilation!)

[Stefan Monnier]

>> Hmm... circularity is quite normal in data structures, yes.
>> But presumably this is only applied to source code where circularity is
>> very rare.  Could it be that you end up recursing in elements which
>> actually aren't part of the source code (and hence can't have
>> symbols-with-positions)?
> I honestly don't know at the moment.

I think it's worth the effort to try and track this down.  Maybe we can
completely circumvent the problem.

>> Also, I see in `byte-compile-strip-s-p-1` that you only look inside conses
>> and vectors.  So I'm not sure what makes you say the recursion was in
>> records since records are similar to vectors but aren't `vectorp` so
>> AFAICT your code won't recurse into them.
>
> byte-compile-strip-s-p-1 has been enhanced to handle records, too,
> though I haven't committed that bit yet (along with quite a lot of other
> amendments, too).

Hmm... now that I think about it, you only generate
symbols-with-positions (symposes) when byte-compiling, right?
And you can restrict this to the case where we byte-compile into a file
(as opposed to the rare case where we just call `byte-compile`).
So the symposes can end up in 2 places:
- in the .elc file: no need to strip the pos here, just make sure the
  symbols get printed without their position.
- elsewhere: that's the problematic part because this only occurs where
  the source code gets stealthy passed elsewhere, e.g. when a macro
  calls (put ARG1 'foo ARG2) during the macro expansion (rather than
  returning that chunk of code in the expansion).  Here we don't have
  much control over where the symposes end up and I don't think
  `byte-compile-strip-s-p` can help us (unless we call it before passing
  the result to the macro, but I don't think that's what we want to do).

So where/why do we need `byte-compile-strip-s-p`?

>> That's what we do elsewhere, yes, except that history taught us that
>> a hash-table is a better choice to avoid scalability problems.
>> Tho in your case you'd only need to keep the stack of objects inside of
>> which you're currently recursing, so maybe a list is good enough.
> I've tried the list approach (using memq to check for an already
> processed cons/vector/record.  It fell flat on its face with
> lisp/leim/ja-dic/ja-dec.el, which has a list with over 60,000 strings
> in it.

Oh, right, we have to add to the list all the conses rather than only
the head conses, so you definitely want to use a hash-table.


        Stefan

Re: Circular records: how do I best handle them? (The new correct warning position branch now bootstraps in native compilation!)

[Alan Mackenzie]

Hello, Stefan.

On Sun, Dec 26, 2021 at 15:35:35 -0500, Stefan Monnier wrote:
> >> Hmm... circularity is quite normal in data structures, yes.
> >> But presumably this is only applied to source code where circularity is
> >> very rare.  Could it be that you end up recursing in elements which
> >> actually aren't part of the source code (and hence can't have
> >> symbols-with-positions)?
> > I honestly don't know at the moment.

> I think it's worth the effort to try and track this down.  Maybe we can
> completely circumvent the problem.

I don't think there are any such cases.  I'll think it through fully,
some time.

> >> Also, I see in `byte-compile-strip-s-p-1` that you only look inside conses
> >> and vectors.  So I'm not sure what makes you say the recursion was in
> >> records since records are similar to vectors but aren't `vectorp` so
> >> AFAICT your code won't recurse into them.

> > byte-compile-strip-s-p-1 has been enhanced to handle records, too,
> > though I haven't committed that bit yet (along with quite a lot of other
> > amendments, too).

> Hmm... now that I think about it, you only generate
> symbols-with-positions (symposes) when byte-compiling, right?

Correct.

> And you can restrict this to the case where we byte-compile into a file
> (as opposed to the rare case where we just call `byte-compile`).

I suppose this could be done, but there's no need.  compile-defun isn't
that rare a function, and we want the correct warning messages from it.

> So the symposes can end up in 2 places:
> - in the .elc file: no need to strip the pos here, just make sure the
>   symbols get printed without their position.

The positions get stripped before the code is dumped to the .elc.

> - elsewhere: that's the problematic part because this only occurs where
>   the source code gets stealthy passed elsewhere, e.g. when a macro
>   calls (put ARG1 'foo ARG2) during the macro expansion (rather than
>   returning that chunk of code in the expansion).

This isn't a problem.  If it is a compiled macro doing this, the
positions will already be gone from the symbols.  If it is from an
uncompiled macro, XSYMBOL in Feval's subroutines does the Right Thing.

>   Here we don't have much control over where the symposes end up and I
>   don't think `byte-compile-strip-s-p` can help us (unless we call it
>   before passing the result to the macro, but I don't think that's
>   what we want to do).

> So where/why do we need `byte-compile-strip-s-p`?

It's now become macroexp-strip-symbol-position, so that it is always
loaded early, and there is no need for a duplicate function in
cl-macs.el any more.  There didn't seem to be a better place to put it.

It's used all over the place.  In eval-when/and-compile, it is used
before the evaluation.  It is used before dumping the byte compiled code
to the file.elc, and before passing this code to the native compiler.
Several (?most) of the byte-compile-file-form-... functions use it.
It's used in the newish keymap functions near the end of bytecomp.el, in
byte-compile-annotate-call-tree, etc.  Also in cl-define-compiler-macro,
and internal-macro-expand-for-load.  Additionally, also from Fput, to
prevent symbols with positions getting into symbol property lists.

> >> That's what we do elsewhere, yes, except that history taught us that
> >> a hash-table is a better choice to avoid scalability problems.
> >> Tho in your case you'd only need to keep the stack of objects inside of
> >> which you're currently recursing, so maybe a list is good enough.
> > I've tried the list approach (using memq to check for an already
> > processed cons/vector/record.  It fell flat on its face with
> > lisp/leim/ja-dic/ja-dec.el, which has a list with over 60,000 strings
> > in it.

> Oh, right, we have to add to the list all the conses rather than only
> the head conses, so you definitely want to use a hash-table.

Yes, I have to do this.  I am still debating whether just to do it
(which might slow things down quite a bit), or to do it in a
condition-case handler after the recursion has exceeded the 1,600
max-lisp-eval-depth.  I'm inclined towards the latter at the moment.

For other Lisp objects with a read syntax, such as char tables and
decorated strings, I intend to amend the reader just to output plain
symbols for them.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Circular records: how do I best handle them? (The new correct warning position branch now bootstraps in native compilation!)

[Stefan Monnier]

>> >> Hmm... circularity is quite normal in data structures, yes.
>> >> But presumably this is only applied to source code where circularity is
>> >> very rare.  Could it be that you end up recursing in elements which
>> >> actually aren't part of the source code (and hence can't have
>> >> symbols-with-positions)?
>> > I honestly don't know at the moment.
>> I think it's worth the effort to try and track this down.  Maybe we can
>> completely circumvent the problem.
> I don't think there are any such cases.

Hmm... I thought this whole circular records thread started because you
bumped into such a case.  I feel like I'm misunderstanding something.

>> So the symposes can end up in 2 places:
>> - in the .elc file: no need to strip the pos here, just make sure the
>>   symbols get printed without their position.
> The positions get stripped before the code is dumped to the .elc.

Why bother?  You can just have a `print-symbols-without-position` which
you let-bind around the printing code.

>> - elsewhere: that's the problematic part because this only occurs where
>>   the source code gets stealthy passed elsewhere, e.g. when a macro
>>   calls (put ARG1 'foo ARG2) during the macro expansion (rather than
>>   returning that chunk of code in the expansion).
> This isn't a problem.  If it is a compiled macro doing this, the
> positions will already be gone from the symbols.  If it is from an
> uncompiled macro, XSYMBOL in Feval's subroutines does the Right Thing.

I didn't mean sympos coming from the macro but sympos coming from the
args passed to the macro.  Something like:

    (defmacro foobar-really (arg1 arg2)
      (puthash arg1 arg2 foobar-remember)
      `(progn (do-something ,arg1) (do-something-else ,arg2)))

The `remember` property will end up containing symbols-with-pos if
`arg2` contains symbols.

> It's used all over the place.  In eval-when/and-compile, it is used
> before the evaluation.  It is used before dumping the byte compiled code
> to the file.elc, and before passing this code to the native compiler.
> Several (?most) of the byte-compile-file-form-... functions use it.
> It's used in the newish keymap functions near the end of bytecomp.el, in
> byte-compile-annotate-call-tree, etc.  Also in cl-define-compiler-macro,
> and internal-macro-expand-for-load.

Interesting.  Why do you need it at so many places?
What is it usually used for?

> Additionally, also from Fput, to prevent symbols with positions
> getting into symbol property lists.

IIUC this is for the kind of example I showed above (tho I used
`puthash` instead of `put`)?

> Yes, I have to do this.  I am still debating whether just to do it
> (which might slow things down quite a bit), or to do it in a
> condition-case handler after the recursion has exceeded the 1,600
> max-lisp-eval-depth.  I'm inclined towards the latter at the moment.

Using a (weak) hash-table may actually speed things up if you call it
from lots and lots of places and it thus ends up being applied several
times (redundantly) to the same data.

> For other Lisp objects with a read syntax, such as char tables and
> decorated strings, I intend to amend the reader just to output plain
> symbols for them.

Sounds reasonable.


        Stefan

Re: Circular records: how do I best handle them? (The new correct warning position branch now bootstraps in native compilation!)

[Alan Mackenzie]

Hello, Stefan.

On Thu, Dec 30, 2021 at 13:37:47 -0500, Stefan Monnier wrote:
> >> >> Hmm... circularity is quite normal in data structures, yes.
> >> >> But presumably this is only applied to source code where circularity is
> >> >> very rare.  Could it be that you end up recursing in elements which
> >> >> actually aren't part of the source code (and hence can't have
> >> >> symbols-with-positions)?
> >> > I honestly don't know at the moment.
> >> I think it's worth the effort to try and track this down.  Maybe we can
> >> completely circumvent the problem.
> > I don't think there are any such cases.

> Hmm... I thought this whole circular records thread started because you
> bumped into such a case.  I feel like I'm misunderstanding something.

What I bumped into was circularly linked vectors in the source code
being compiled.

I've amended the reader so that it doesn't put positions on symbols
which are read as components of other structures such as byte compiled
functions, text property lists in strings, and so on.  (Actually, there
was very little to amend.).

I've amended macroexp-strip-symbol-positions so that it ignores
circularity unless it hits an infinite recursion, in which case it
starts again, recording all components found in hash tables.

I committed these changes a short time ago.

> >> So the symposes can end up in 2 places:
> >> - in the .elc file: no need to strip the pos here, just make sure the
> >>   symbols get printed without their position.
> > The positions get stripped before the code is dumped to the .elc.

> Why bother?  You can just have a `print-symbols-without-position` which
> you let-bind around the printing code.

I think I've got that already, though it's a long time since I looked at
it.

> >> - elsewhere: that's the problematic part because this only occurs where
> >>   the source code gets stealthy passed elsewhere, e.g. when a macro
> >>   calls (put ARG1 'foo ARG2) during the macro expansion (rather than
> >>   returning that chunk of code in the expansion).
> > This isn't a problem.  If it is a compiled macro doing this, the
> > positions will already be gone from the symbols.  If it is from an
> > uncompiled macro, XSYMBOL in Feval's subroutines does the Right Thing.

> I didn't mean sympos coming from the macro but sympos coming from the
> args passed to the macro.  Something like:

>     (defmacro foobar-really (arg1 arg2)
>       (puthash arg1 arg2 foobar-remember)
>       `(progn (do-something ,arg1) (do-something-else ,arg2)))

Args which are symbols with positions are first and foremost symbols.
They behave like symbols when used.  The test of this is that Emacs
bootstraps, despite having many macro expressions like ,@body which
expand to expressions with positions.

> The `remember` property will end up containing symbols-with-pos if
> `arg2` contains symbols.

In that (puthash arg1 arg2 foobar-remember), if the key is a symbol with
position, it is stripped.  I think the value will keep its positions.
This might still be a problem.

> > It's used all over the place.  In eval-when/and-compile, it is used
> > before the evaluation.  It is used before dumping the byte compiled code
> > to the file.elc, and before passing this code to the native compiler.
> > Several (?most) of the byte-compile-file-form-... functions use it.
> > It's used in the newish keymap functions near the end of bytecomp.el, in
> > byte-compile-annotate-call-tree, etc.  Also in cl-define-compiler-macro,
> > and internal-macro-expand-for-load.

> Interesting.  Why do you need it at so many places?
> What is it usually used for?

Stipping positions from compiled code before dumping it to an .elc file,
and also before passing the compiled code to the native compiler.

The fact is these positions on the symbols are unwanted for most uses of
symbols around compilation, being needed only in the analysis phase of
the source code.

> > Additionally, also from Fput, to prevent symbols with positions
> > getting into symbol property lists.

> IIUC this is for the kind of example I showed above (tho I used
> `puthash` instead of `put`)?

> > Yes, I have to do this.  I am still debating whether just to do it
> > (which might slow things down quite a bit), or to do it in a
> > condition-case handler after the recursion has exceeded the 1,600
> > max-lisp-eval-depth.  I'm inclined towards the latter at the moment.

> Using a (weak) hash-table may actually speed things up if you call it
> from lots and lots of places and it thus ends up being applied several
> times (redundantly) to the same data.

In the end I went with the condition-case approach, based on a gut
feeling that hash tables aren't very fast, and they're needed only
rarely, certainly whilst bootstrapping Emacs.

> > For other Lisp objects with a read syntax, such as char tables and
> > decorated strings, I intend to amend the reader just to output plain
> > symbols for them.

> Sounds reasonable.

I've done this now.  Feel free to look at the new version of
scratch/correct-warning-pos.

And a Happy New Year!

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Circular records: how do I best handle them? (The new correct warning position branch now bootstraps in native compilation!)

[Stefan Monnier]

>> >> >> Hmm... circularity is quite normal in data structures, yes.
>> >> >> But presumably this is only applied to source code where circularity is
>> >> >> very rare.  Could it be that you end up recursing in elements which
>> >> >> actually aren't part of the source code (and hence can't have
>> >> >> symbols-with-positions)?
>> >> > I honestly don't know at the moment.
>> >> I think it's worth the effort to try and track this down.  Maybe we can
>> >> completely circumvent the problem.
>> > I don't think there are any such cases.
>> Hmm... I thought this whole circular records thread started because you
>> bumped into such a case.  I feel like I'm misunderstanding something.
> What I bumped into was circularly linked vectors in the source code
> being compiled.

Then my question above turns into: what is this source code?

> I've amended the reader so that it doesn't put positions on symbols
> which are read as components of other structures such as byte compiled
> functions, text property lists in strings, and so on.  (Actually, there
> was very little to amend.).

OK.

>> > The positions get stripped before the code is dumped to the .elc.
>> Why bother?  You can just have a `print-symbols-without-position` which
>> you let-bind around the printing code.
> I think I've got that already, though it's a long time since I looked at
> it.

So why do you need to strip the positions before dumping the code into
the `.elc`?

>>     (defmacro foobar-really (arg1 arg2)
>>       (puthash arg1 arg2 foobar-remember)
>>       `(progn (do-something ,arg1) (do-something-else ,arg2)))
[...]
> In that (puthash arg1 arg2 foobar-remember), if the key is a symbol with
> position, it is stripped.  I think the value will keep its positions.
> This might still be a problem.

`put` and `puthash` are just some of the ways a macro's arg can
"escape".  A macro may also something like

    (push arg my-list-of-stuff)

Having to strip symbol positions in `put` and `puthash` (i.e. having
this implementation detail leak to those places which aren't directly
related to compilation) is pretty ugly.  Do we really want to extend
that to `setq`, `aset`, and whatnot?

Maybe we should "bite the bullet" and expect macros to announce whether
they support sympos or not and if they don't we strip the positions
before calling them (we can try to be a bit more clever by using the
Edebug spec to find args where sympos will probably be harmless).

>> > It's used all over the place.  In eval-when/and-compile, it is used
>> > before the evaluation.  It is used before dumping the byte compiled code
>> > to the file.elc, and before passing this code to the native compiler.
>> > Several (?most) of the byte-compile-file-form-... functions use it.
>> > It's used in the newish keymap functions near the end of bytecomp.el, in
>> > byte-compile-annotate-call-tree, etc.  Also in cl-define-compiler-macro,
>> > and internal-macro-expand-for-load.
>> Interesting.  Why do you need it at so many places?
>> What is it usually used for?
> Stipping positions from compiled code before dumping it to an .elc file,

That sounds like "one place"

> and also before passing the compiled code to the native compiler.

And that sounds like "a second place".
In contrast above you list all kinds of *other* places.
Why do we need to strip positions in those other places?
Could we instead change the code (e.g. byte-compile-annotate-call-tree)
so it works with sympos?

> The fact is these positions on the symbols are unwanted for most uses of
> symbols around compilation, being needed only in the analysis phase of
> the source code.

So you're saying the problem is that your compiler doesn't separate the
front end from the backend?  That's indeed an inconvenient of the
current bytecompiler code.



        Stefan

Re: Circular records: how do I best handle them? (The new correct warning position branch now bootstraps in native compilation!)

[Alan Mackenzie]

Hello, Stefan.

On Sat, Jan 01, 2022 at 12:31:51 -0500, Stefan Monnier wrote:

[ .... ]

> > What I bumped into was circularly linked vectors in the source code
> > being compiled.

> Then my question above turns into: what is this source code?

> > I've amended the reader so that it doesn't put positions on symbols
> > which are read as components of other structures such as byte compiled
> > functions, text property lists in strings, and so on.  (Actually, there
> > was very little to amend.).

> OK.

> >> > The positions get stripped before the code is dumped to the .elc.
> >> Why bother?  You can just have a `print-symbols-without-position` which
> >> you let-bind around the printing code.
> > I think I've got that already, though it's a long time since I looked at
> > it.

> So why do you need to strip the positions before dumping the code into
> the `.elc`?

Thank you very much indeed for this tip.  I don't need to strip the
positions.  eval already handles symbols with position (provided
symbols-with-pos-enabled is non-nil), as does pretty much everything
else, including the native-compiler.

Binding that variable and print-symbols-bare to non-nil rather than
stripping positions was actually quite simple, compared with the mess I
was in trying to deal with the circularity in some of the
lists/vectors/records.  I profiled some of the compilation runs with the
stripping strategy, and garbage collection was consuming around 70% of
the run time.  :-(

I've now got the thing working modulo tidying up.  A make bootstrap now
takes 7min 45sec on my machine, compared with 7min 18sec for the same on
the master branch.  That's a 7% difference.  However, I've still got to
strip out the old warning position mechanism, which should shave
something off of that 7% difference.

[ .... ]

> `put` and `puthash` are just some of the ways a macro's arg can
> "escape".  A macro may also something like

>     (push arg my-list-of-stuff)

> Having to strip symbol positions in `put` and `puthash` (i.e. having
> this implementation detail leak to those places which aren't directly
> related to compilation) is pretty ugly.  Do we really want to extend
> that to `setq`, `aset`, and whatnot?

No.  What we have to do is NOT to strip positions off of these objects,
instead warning users to be careful about saving bits of code in a way
that survives the byte compilation.  Possibly we should give them the
position stripping function to use at their discretion.  What do you
think?

[ .... ]

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).