bug#67455: (Record source position, etc., in doc strings, and use this in *Help* and backtraces.)

[Alan Mackenzie <acm@muc.de>]

Hello, Stefan.

On Tue, Mar 26, 2024 at 16:30:06 -0400, Stefan Monnier wrote:
> > We now have two distinct uses of SWPs: providing warning source locations
> > to the compiler (where we want to keep the position as long as possible)
> > and providing position information for the doc string (where we want to
> > strip the position from the symbol ASAP, to avoid trying to use the SWP
> > when we need a plain symbol).  If both of these occur together, we want
> > to keep the SWP.

> I think I'm beginning to understand.  So in the "load from source case",
> some of your symbols are SWPs and you want to turn them into bare
> symbols "on the fly" during macro-expansion rather than via a separate
> "strip" phase, ....

More precisely, to @dfn{posify} their  containing forms by writing the
position information into their doc strings.  We do this in the byte
compilation case, too.  The difference is that in the "load from source"
case we want to strip the SWP, in byte compilation, we don't.

> .... so you want the macro expansion to know whether it's done
> for "load from source" or for some other purpose and you use
> `byte-compile-in-progress` as a proxy for that information.
> Is that it?

More or less.  But byte-compile-in-progress isn't a proxy, it's the prime
criterion for deciding.

> If so, (and if the stripping happens within macros), then indeed passing
> it as a separate argument through all the recursive calls to
> `macroexp--expand-all` would be cumbersome.  But I suggest you use
> another name for that(e.g. `macroexp-strip-position`) so the
> intention is made more clear.

I don't think that is a good name.  The byte compiler has no business
setting "internal" variables for the posification processing.  Instead it
should announce it's running and expect the posification to respect that.
I think byte-compile-in-progress is a good name for this.

> Better yet: to avoid the problem of dynamic scope extending "too far"
> (i.e. accidentally applying to nested loads/evals/compile/...), you
> could put the relevant info into `macroexpand-all-environment`.
> [ That var is also dynamically bound, but we're already careful to
>   rebind it when needed so it doesn't apply to nested uses
>   of macroexpansion.  ]

That variable is only loaded in the 17th loaded Lisp file.  The new
facility should be working at the earliest stages of loading Lisp, as it
does at the moment.  Besides, macroexpand-all-environment is not
documented anywhere, what it is, what it's for, etc.

> >> > By "currently", I mean that a defining form such as defun or defvar has
> >> > commenced, but not yet terminated; its functions currently occupy stack
> >> > frames.
> >> So you mean we're inside `Fdefalias` or `Fdefvar_1`?
> > Yes, or inside a macro (defun, defmacro, ...) which expands to a
> > defalias.

> These are *very* different times: `Fdefalias` and `Fdefvar_1` are
> executed long after macroexpansion.  And they're small C functions which
> run almost no external code at all, so they "occupy stack frames" only
> for a very short time.

I think we were talking about the handling of defining-symbol.  It has a
valid binding outside of macros such as defun, and this binding is used
to posify the containing form.

> My crystal ball suggests that "currently" may be the wrong way to think
> about it: maybe instead of thinking of "when" (as in "during the
> definition of function FOO") what you're looking for might be "where"
> (as in "within the body of FOO").
> [ That's the same difference as the difference between dynamic and
>   static scoping.  ]

I'm having trouble understanding what you're saying, here.

> If my crystal ball is right, then the better place to put that info is
> probably `macroexpand-all-environment`.

See above.

> > Ideally, I would like to have bound defining-symbol inside defun.

>     (defmacro my-defun (name args &rest body)
>       `(cl-macrolet ((defining-symbol () '',name))
>         (defun ,name ,args ,@body)))

>     (my-defun my-foo (x) (list x (defining-symbol)))

>     (symbol-function 'my-foo)
>     ==> #f(lambda (x) [t] (list x 'my-foo))

> `cl-macrolet` uses `macroexpand-all-environment` for that.

cl-macs gets loaded far too late for such an approach to be useful.

> > Fload uses read-positioning-DEFINED-symbols, as contrasted with the
> > compiler, which uses read-positioning-symbols.  r-p-d-s positions only
> > lambdas and NAMEs.  r-p-s positions all symbols except nil.

> I think I'm beginning to understand (I guess I was struggling with your
> use of "position" as a verb for some reason which made me think that
> symbols were being moved rather than gaining position information).

Sorry about that.

> So, IIUC you use `read-positioning-DEFINED-symbols` instead of
> `read-positioning-symbols` because it's cheaper?

No, because it does the Right Thing.

> Do you have rough numbers comparing the cost of `read`,
> `read-positioning-symbols`, and `read-positioning-DEFINED-symbols`?

No, but they will be very close to eachother (and very cheap) since they
use the same engine, read0 (in lread.c).  Each of them will be one or two
orders of magnitude faster than emulating them in Lisp.

> Also, IIUC you don't have a separate phase to strip the SWPs when
> loading from source, but instead you strip them as you "consume" their
> info during macroexpansion.  If so, how/when/where do you strip the
> false positives that may occur inside quoted data or in code like:

>     (defmacro foo (lambda bar) ...)
>     (defmacro foo (defun bar) ...)

>     (let* ((lambda foo)
>            (defun bar))
>       ...)

There's a pcase arm right at the end of macroexp--expand-all which strips
SWPs of their positions.  Recursing through macroexp--all-forms will
eventually hit this pcase arm for these lambdas.

> > Ah, right.  I hadn't considered this before.  The changes are by their
> > very nature essentially complicated and difficult to understand.

> [ Hmm... maybe not the best salespitch.  ]

;-)  It's the truth, though.

> >> I think you can simply wait to add the entry to
> >> `macro-declarations-alist` until a later time, so the `defining-symbol`
> >> thingies will be ignored during the early bootstrap and once we have
> >> more infrastructure in place we can then register the handler on
> >> `macro-declarations-alist`.

> > This will not be simpler.  It would involve re-evaluating defun, then
> > compensating for all the functions up to now whose NAMEs had been read
> > without positions.

> Not at all.  Those will remain without position, but only in
> `src/bootstrap-emacs`.

This would be a Bad Thing.  The current code is active right after
loading byte-run.

> In the real `src/emacs` they will get the position because they'll come
> from the `.el[cn]` file and by the time we get compile those files
> `macro-declarations-alist` will be fully populated.

The understanding we reached in November was that loading from source
files would be handled, too.

> > There is unavoidable conplexity, here.

> I'm definitely not convinced.  I suspect you've been asking yourself
> "can it be made simpler" and you may indeed then convince yourself that
> the answer is no, because of assumptions you don't reconsider.

> Try instead to think about "what would it take to remove that complexity?".

> > I see things somewhat differently.  We shouldn't increase the debugging
> > burden even on "expert users".

> Yet it's imposing more complexity (and hence more debugging burden) on
> those same expert users.  🙁

That's a fairly difficult philosophical question.  Do we provide full
functionality at the cost of more (difficult) source code, or do we
restrict the functioality to keep the source simpler?  I think with Emacs
we usually go with the first alternative.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).