bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Alan Mackenzie <acm@muc.de>]

Hello, Stefan.

On Sun, Oct 29, 2023 at 12:32:32 -0400, Stefan Monnier wrote:
> >> Hmm... it seems like keeping the LINE+COL info rather than
> >> a surrounding definition would be better in this regard since it wouldn't
> >> require any changes at all to the actual lambdas, regardless where
> >> they are.
> > I don't agree with this.  Whether we add FILE+LINE+COL info is to a
> > large degree a separate question from that of adding defining symbol
> > info.

> It seems to me, if you have one of the two, the other becomes much less
> important.

True, but still important nevertheless.

> > If you don't add information to the actual lambda, then that information
> > won't be available when that lambda turns up in a backtrace or as a
> > value in C-h v.  Surely you'd have to keep FILE+LINE+COL in the lambda
> > itself, somehow.

> In the docstring.

But the docstring isn't in the lambda at the moment.  There's merely a
pointer to it there, which is no good for backtraces - it's too fragile.

> > It will also stress the user somewhat, since it will force her to look
> > up a file location to get any idea of what is failing.  A defining
> > symbol is much more immediately useful, even if it may not be 100%
> > dependable because of the issues you raise below.

> ??

Dealing with backtraces is a stressful business.  Forcing a user to look
somewhere else for pertinent information can only increase that level of
stress.

> >> The notion of "surrounding definition" is less precise (there can be
> >> several lambdas inside a given definition, there can be lambdas outside
> >> of any definition, ...).
> > Yes, there can be several lambdas in a function, though I don't think
> > this is common.

> You'd be surprised, especially with lambdas introduced by macros.

> > Certainly it hasn't given me any problems in use, so far.

> In the past, the byte-compiler only provided the name of the surrounding
> definition in its warnings.  Maybe we should go back to that?  :-)

> > But lambdas outside of definitions?  I doubt very much there are
> > any in the Emacs sources, or even in the packages.

>     grep -C1 '^(add-hook' **/*.el | grep '(lambda'

> begs to differ.

OK, good point!  Here, obviously, the hook symbol should become the
defining symbol, though admittedly this isn't coded up, yet.

> >> Furthermore, that info is already available to the bytecompiler, so it's
> >> actually easier to keep that info.
> > How are you actually going to use this information?  What are you going
> > to print in a backtrace, in particular in a batch mode backtrace where
> > things like buttons are not useful?

> FILE:LINE ?

That is anything but concise, in contrast to printing the defining
symbol in braces.  I've just had a look at a bytecomp.elc, and the
filename in the docstring field is 67 characters long.  In anything
short of a full screen Emacs window, this, together with the name of the
subr, is going to spill over onto the next line.  This is not good.  Or
you have to abbreviate the filename somehow, but this is also unlikely
to be satisfactory.

Besides which, you'd need to store all the pertinent information, the
filename + line + column in RAM, probably in the subr or the byte coded
function.  That's assuming there is a file from which the lambda was
compiled, which there might well not be.  There's another special case.

> > For that matter, what are you going to print in that C-h v example?

> #<subr FILE:LINE HEX>?

> Tho we could skip the FILE:LINE and buttonize some part of the string
> instead, for the `C-h v` case.

You'd have to make a user option to do this, since fragmenting the
information is unkind to those who don't like jumping backwards and
forwards.

> >> Yup.  [ Tho now that you make me think about it, if we want to be cheap,
> >> we could go to that OFFSET position and look around for a `^(defalias`
> >> and that might be sufficient to give us "the surrounding definition" in
> >> many cases :-)  ]
> > That doesn't sound systematic, or at all attractive.  ;-)

> But so much fun!
> Admittedly, I haven't jumped on this hack because my own local hacks
> move the docstrings to the end of the `.elc` files, making this
> additional hack impossible :-)

> >> What I meant is that the docstring is in the `.elc` file.  So the
> >> bytecode object would be literally unchanged, it's just that the
> >> bytes found at OFFSET inside FILE would be extended to hold:
> >> - the docstring
> >> - the human-readable arglist
> >> - the LINE+COL info

> > Don't forget, all this info would have to go into .eln files too,
> > probably in the struct Lisp_Subr.

> That's the beauty of it: by storing the info inside the raw docstring,
> we get to reuse all the existing code that takes care of storing the
> docstring somewhere.

Making another non-obvious structure to compete with the (lexical)
parameter specs.

As I keep on saying, a backtrace may not depend on reading files; the
error it's reporting on could well be in that very mechanism itself.  We
already had a problem once due to the use of cl-print.el in
debug-early.el, and cl-print is relatively simple compared with file
reading.

> Also, the "performance profile" of docstring matches our needs here:
> just like docstrings, we don't need FILE+LINE+COL info under normal use,
> it's only needed when developing/debugging, so we want to store it "out
> of line".

Disagree: anybody getting a backtrace, or looking up a variable with C-h
v needs this information, at the very least for including in a bug
report.  ;-)

> > It would be better if it were also somehow available in uncompiled
> > lambdas, something which is needed for debugging failures in
> > early bootstrapping.

> Yeah, "WIBNI", but as I said, when you have an uncompiled lambda, the
> content of the body is usually sufficient to manually find the
> matching source.

Usually, but not always.  There are some lambdas added to hooks which
are functionally the same as ignore.

> So I think we should focus on the compiled case first and then maybe
> consider how or if we can/should extend it to the uncompiled case.

No.  Uncompiled code is more difficult to instrument (I'm talking from
experience here), and how that gets to be done influences how it can be
done for compiled code.  We cannot ignore it, push it down the road, and
hope that somehow it can be tacked on later.

> [ And, yes, if we really want to, I think we can add the FILE+COL info
> to uncompiled lambdas: it would force us to read `.el` files "with
> sympos" (thus slower) but it shouldn't be hard.  ]

Again, there must be no file reading when constructing a backtrace.  All
the pertinent information must be in the list form of the lambda or
closure at load time.  If the location of this new info is early on in
the function, it must be distinguishable from what used to be there in
the current format.  I've been through all this already.  The use of a
non-nil symbol preceding the parameter list cannot get confused with the
parameter list itself, which is still there in functions we will still
load.

I still believe that the defining symbol scheme is the most practical
way of conveying the needed extra information to the user.  Otherwise I
would have hacked something different.  ;-)

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).