bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>]

>> Bootstrap problems are rare, and difficult to predict, so we fix them in
>> a reactive way.  Debugging a bootstrap is generally more difficult.
> I'm fully aware of this, having spent a significant part of the last few
> years doing just that.  I aim to make it easier, as I have already done
> to a certain degree.

Exactly.  That makes you put much more emphasis on this aspect than it
generally deserves.  *Very* few Emacs users ever do that.

> That doesn't seem relevant to bootstrapping bugs.  Using gdb on part of
> the bootstrap is quite difficult.

GDB is my "go to" tool when I have an error during bootstrap and I find
it fairly easy to use.

>> We shouldn't design the featureset for the corner case of when the
>> bootstrap fails, because it is a very small fraction of the use-cases,
>> that 99.99% of Emacs users never face.
> We should design the feature set to handle _ALL_ cases, including that
> of bootstrapping.  That's why I wrote debug-early.el, for example; it's
> proved its worth, certainly for me personally.

Note that `early-debug.el` doesn't handle "_ALL_" cases: it's only used
for those cases where the normal debugger can't be used.

Before catering to the same use-cases as early-debug, we should cater to
the use cases of the average user/developer who's never getting close to
the bootstrap.

>> Fist, I did not make a claim to the contrary, so the burden of proof is
>> not on me.  This said, I believe there is ample evidence out there,
>> because all the programming tools I've ever used try first and foremost
>> to let you jump to the corresponding source code.  The source code is
>> what programmers are familiar with, so being able to jump from the
>> backtrace to the corresponding source code is generally the best known
>> way to help the user find the pertinent info.
> Even better is if the relevant information is present on the screen or
> in the backtrace without having to jump to the source code.

How are you going to fix the problem without going to the source code?

> This is the point of the defining symbol.  The "jumping" is only
> straightforward from a running Emacs session.  From a batch mode
> backtrace (e.g. from bootstrapping or ERT), it is much
> more cumbersome.

If your batch session's output is piped to an auto-reverted file in
compilation-mode (mine usually is, FWIW), it's small matter of making
sure the format is recognized.

Still, I refuse to change the syntax of `lambda` just to make it
slightly easier to debug the odd bootstrap problem.

> >From the FILE+LINE+COL info the user can recover the info your patch
>> provides, but the reverse is generally not true.  Of course, there is
>> the exception of when the source code has been changed or is missing or
>> somesuch, but I don't think we should consider those cases when
>> designing the featureset either, because here as well, this is expected
>> to be a fringe case.
> I would expect it to be quite common.

From where I stand:

- If you don't have the source code any more, then you don't have anywhere
  to fix the problem.
- If the source code has changed, then maybe the problem has already been
  corrected (or modified) by that change, so there's not much point
  debugging it any further.

So, why would you consider it to be common?
What kind of scenario do you have in mind where this is sufficiently
common to be worth paying much attention to it?

> We clearly differ in our views of what backtraces should do.

I don't know why you insist so much on backtraces.
I've wished for that info in many more cases than backtraces.

> Also, I think the backtrace code should be as robust as possible, you
> don't seem to.

I want it to be featureful, with a fallback for those rare cases when
that's not an option.  You provided the fallback already in `debug-early`.

> I will be working on putting this file and position information into the
> code.  It will be tedious rather than difficult.

Why would it be tedious?  It should be easier than what you did because
you don't need an extra arg to `lambda` and you don't need to
`defining-symbol` any more.  It should make the change simpler.

>> > Which looks like you intend to strip it [debugging instrumentation]
>> > out, leaving interpreted code as a special case without the
>> > instrumentation.
>> Yes.
> OK, we have to disagree on this.  For example typing d in edebug to get
> a backtrace should, in my view, show that info in the backtrace.

I don't find this particularly important.

> Your way would make it unavailable.

You could tweak Edebug to add that info, thus keeping the change localized
to Edebug.

>> But as it currently stands, it comes at some costs:
>> - changing the syntax of `lambda`.
> lambda in the bulk of the sources remains unchanged.  Only for very
> occasional cases has the syntax been enhanced.

Still, it changes the syntax of a very fundamental building block.
That's a pretty high cost in my book compared to the benefit it brings.

If you want to do that, I'd rather you keep `lambda` as before and
macro-expand it to a new form that has the new syntax.

Part of the cost of your change here is that it prevents future changes
that might want to use a similar syntax for other purposes.
IOW it "eats up namespace", tho "namespace" is not quite the right word.

>> - changing the representation of functions (both bytecode and interpreted).
> Don't forget the native compiled, too!

That's more internal and it's currently not used for closures, so I find
this part of the cost negligible.

> Only the change to interpreted code is non-trivial.  For the compiled
> code, there has simply been an extra field inserted at or near the end
> of the function structures.

For bytecode it's still an extra field (which we have to pay for every
closure we create) thus increasing memory use.  Plus the incompatible
change of position of the interactive form (tho I find this less of an
problem).

Maybe in your tests, the increase is negligible, but remember: every
user will pay this tiny cost, and most of the users will never reap any
benefit from it, and those who do will only do very rarely.

It's a good change for us hard-core Emacs hackers, but it has to come
*very* cheap because the gain is really low for other users.

>> Storing the info in the docstring would fix the second and third point.
>> Using the FILE+LINE+COL info would fix the first.
> I don't know what you mean by "docstring".

Docstrings live in the `.elc` files.

> You could mean either in the ..elc/.eln file or in the function
> structure or both.  I don't see docstring storage as a very good idea.

Probably because you think of it as a docstring only.
Think of it as "out of line storage" where we can keep introspection
data for which we don't want to pay *any* cost (other than disk space
use) in the normal execution of code (i.e. when calling the closure or
when creating the closure) such as:
- documentation
- human-readable arglist
- source code location
- debug info
- a copy of the source code
- ...

> It seems like a kludge, and would complicate Emacs's handling, much as
> the existing new code complicates it.

I see it as an opportunity to clean up this kludge.


        Stefan