bug#67455: (Record source position, etc., in doc strings, and use this in *Help* and backtraces.)

[Alan Mackenzie <acm@muc.de>]

Hello, Stefan.

On Thu, Mar 28, 2024 at 12:25:11 -0400, Stefan Monnier wrote:

[ .... ]

> >> >> Do you have rough numbers comparing the cost of `read`,
> >> >> `read-positioning-symbols`, and `read-positioning-DEFINED-symbols`?
> >> > No, but they will be very close to eachother (and very cheap)
> >> Then I think we should use `read-positioning-symbols`, which
> >> requires fewer code changes.
> > It won't.  Required would be Lisp code to determine whether a particular
> > SWP needs to be stripped or not.  This is not going to be simple.  It is
> > likely to be about as complicated as the existing enhancements to read0.

> They'd all need to be stripped, AFAICT, so we'd do:

>     (strip-all-symbol-positions
>      (macroexp--expand-all
>       (read-positioning-symbols)))

> What would be hard about it?

Some symbols must not be stripped.  For example, in cl-generic.el L403
we have:

    (fun `(cl-function (lambda ,plain-args ,@body)))              

..  There the position on the lambda must be preserved until ME2 time
when it becomes clear what the shape of the lambda is.  In particular,
whether there is already a doc string in ,@body to amend, or we need to
insert a new one.

> >> >> Also, IIUC you don't have a separate phase to strip the SWPs when
> >> >> loading from source, but instead you strip them as you "consume" their
> >> >> info during macroexpansion.  If so, how/when/where do you strip the
> >> >> false positives that may occur inside quoted data or in code like:

> >> >>     (defmacro foo (lambda bar) ...)

(defmacro foo (lambda bar)
  `(cons ,lambda ,bar))

exoands to

(macro closure (t) (lambda bar) ";POS^^^A^A^A [foo *scratch* 158 nil]
" (list 'cons lambda bar))

..

> >> >>     (defmacro foo (defun bar) ...)

> >> >>     (let* ((lambda foo)
> >> >>            (defun bar))
> >> >>       ...)

Similarly, we get

(defun baz ()
  (let ((lambda 'foo)
        (defun 'bar))
    (cons lambda defun)))

(symbol-function 'baz)
(closure (t) nil ";POS^^^A^A^A [baz *scratch* 323 nil]
" (let ((lambda 'foo) (defun 'bar)) (cons lambda defun)))

, so it is clear this case is getting handled OK.  I'm afraid I can't
point out the exact place in the code at the moment where this is
getting done.

> >> > There's a pcase arm right at the end of macroexp--expand-all which strips
> >> > SWPs of their positions.  Recursing through macroexp--all-forms will
> >> > eventually hit this pcase arm for these lambdas.

> Actually, now that I look at the code I only see:

>             ((guard (and (not byte-compile-in-progress)
>                          (symbol-with-pos-p form)))
>              (bare-symbol form))

> is that the "arm" you're talking about?

Yes.

> AFAICT this will handle only those symbols which appear as Lisp
> expressions (IOW symbols which are variable references), so it will
> strip the `bar` in the second example but not the `bar` in my first
> exmple, nor the two `lambda`s, nor those in

>     '(lambda (defun bar))

See above.

[ .... ]

> > Why do you think this design change will be better than the existing
> > design?

> I don't actually know whether it will be better.  It just seems it could
> lead to simpler code, with no change at all to the reader, for example.

The exercise is intrinsically complicated.  read0 is largely self
contained, meaning any compexity introduced there won't spill over into
other functions.  What you're suggesting is that the code to decide
which SWPs to strip is going to be simpler than the enhancements to the
reader.

> I'm here asking what lead you to the current design, under the
> assumption that the complexity you introduced was the result of
> other experiments.

The complexity is essential to the task being done.  I'm convinced it
cannot be avoided, though it could be moved around the code base, to
some extent.  I don't have a record of the precise reasons for the
design.  To a large extent it was a sequence of solutions to the often
awkward problems the code presented.

> Am I to understand that the current code is basically your first attempt
> at adding such functionality?

By no means.  This is the second implementation, the first having been
rejected by Eli and you last autumn.  I'm very much looking forward to
not needing a third implementation.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).