bug#66750: Unhelpful text in C-h v for variables with a lambda form as value

[Alan Mackenzie <acm@muc.de>]

Hello, Stefan.

On Wed, Nov 01, 2023 at 14:11:33 -0400, Stefan Monnier wrote:
> >> Bootstrap problems are rare, and difficult to predict, so we fix them in
> >> a reactive way.  Debugging a bootstrap is generally more difficult.
> > I'm fully aware of this, having spent a significant part of the last few
> > years doing just that.  I aim to make it easier, as I have already done
> > to a certain degree.

> Exactly.  That makes you put much more emphasis on this aspect than it
> generally deserves.  *Very* few Emacs users ever do that.

That's just your view.  You're wrong, here.  I've put little emphasis on
bootstrap problems.  That they are handled is a side effect of handling
everything.

> > That doesn't seem relevant to bootstrapping bugs.  Using gdb on part of
> > the bootstrap is quite difficult.

> GDB is my "go to" tool when I have an error during bootstrap and I find
> it fairly easy to use.

OK, I found it very difficult to get a working command line for this.
Maybe if you do it often enough it becomes second nature.

> >> We shouldn't design the featureset for the corner case of when the
> >> bootstrap fails, because it is a very small fraction of the use-cases,
> >> that 99.99% of Emacs users never face.
> > We should design the feature set to handle _ALL_ cases, including that
> > of bootstrapping.  That's why I wrote debug-early.el, for example; it's
> > proved its worth, certainly for me personally.

> Note that `early-debug.el` doesn't handle "_ALL_" cases: it's only used
> for those cases where the normal debugger can't be used.

That's twisted reasoning.  early-debug.el will handle anything that gets
thrown at it.

> Before catering to the same use-cases as early-debug, we should cater to
> the use cases of the average user/developer who's never getting close to
> the bootstrap.

I disagree.  I don't understand why you want to split off (some of) "the
use cases of the average user" and only handle those cases.  My
intention is to handle _ALL_ cases, and you find this objectionable for
some reason.

> >> Fist, I did not make a claim to the contrary, so the burden of proof is
> >> not on me.  This said, I believe there is ample evidence out there,
> >> because all the programming tools I've ever used try first and foremost
> >> to let you jump to the corresponding source code.  The source code is
> >> what programmers are familiar with, so being able to jump from the
> >> backtrace to the corresponding source code is generally the best known
> >> way to help the user find the pertinent info.
> > Even better is if the relevant information is present on the screen or
> > in the backtrace without having to jump to the source code.

> How are you going to fix the problem without going to the source code?

You're being contrary again.  My intention is that the problem should be
better understood before going to the source code - in fact better to
understand which source code is relevant.

> > This is the point of the defining symbol.  The "jumping" is only
> > straightforward from a running Emacs session.  From a batch mode
> > backtrace (e.g. from bootstrapping or ERT), it is much
> > more cumbersome.

> If your batch session's output is piped to an auto-reverted file in
> compilation-mode (mine usually is, FWIW), it's small matter of making
> sure the format is recognized.

In other words, assume the user is advanced enough to go through all
these motions, just to be able to work the way you do.  I suspect most
users will not be doing this.  I certainly don't (at least, not yet).

> Still, I refuse to change the syntax of `lambda` just to make it
> slightly easier to debug the odd bootstrap problem.

For crying out loud, I've explained to you often enough, the syntax of
lambda is _NOT_, I repeat _NOT_ being changed.

> > >From the FILE+LINE+COL info the user can recover the info your patch
> >> provides, but the reverse is generally not true.  Of course, there is
> >> the exception of when the source code has been changed or is missing or
> >> somesuch, but I don't think we should consider those cases when
> >> designing the featureset either, because here as well, this is expected
> >> to be a fringe case.
> > I would expect it to be quite common.

> >From where I stand:

> - If you don't have the source code any more, then you don't have anywhere
>   to fix the problem.
> - If the source code has changed, then maybe the problem has already been
>   corrected (or modified) by that change, so there's not much point
>   debugging it any further.

> So, why would you consider it to be common?

Because people change source code, without necessarily compiling it
straight away.  Maybe you don't.  Other people do.

> What kind of scenario do you have in mind where this is sufficiently
> common to be worth paying much attention to it?

Wrong question.  I've already explained enough, I believe backtraces and
things like C-h v should work well IN ALL CASES.  You don't.  I'm asking
you to accept this difference in view between us.

> > We clearly differ in our views of what backtraces should do.

> I don't know why you insist so much on backtraces.
> I've wished for that info in many more cases than backtraces.

That info will get displayed by anything using cl-prin1.

> > Also, I think the backtrace code should be as robust as possible, you
> > don't seem to.

> I want it to be featureful, with a fallback for those rare cases when
> that's not an option.  You provided the fallback already in `debug-early`.

But the information it currently prints is inadequate for lambda forms.
That's what I've fixed, and it will be fixed in the "normal" backtrace
code, too.  You don't like it, for some reason.

> > I will be working on putting this file and position information into the
> > code.  It will be tedious rather than difficult.

> Why would it be tedious?  It should be easier than what you did because
> you don't need an extra arg to `lambda` and you don't need to
> `defining-symbol` any more.  It should make the change simpler.

I'm not coding up your design decisions which I believe are misguided.
Perhaps you should code up these things yourself if you think it's going
to be so fascinating.  Then we'll be able to compare my patch with
yours.

> >> > Which looks like you intend to strip it [debugging instrumentation]
> >> > out, leaving interpreted code as a special case without the
> >> > instrumentation.
> >> Yes.
> > OK, we have to disagree on this.  For example typing d in edebug to get
> > a backtrace should, in my view, show that info in the backtrace.

> I don't find this particularly important.

Other people, including me, do.

> > Your way would make it unavailable.

> You could tweak Edebug to add that info, thus keeping the change localized
> to Edebug.

I doubt it.

> >> But as it currently stands, it comes at some costs:
> >> - changing the syntax of `lambda`.
> > lambda in the bulk of the sources remains unchanged.  Only for very
> > occasional cases has the syntax been enhanced.

> Still, it changes the syntax of a very fundamental building block.
> That's a pretty high cost in my book compared to the benefit it brings.

You don't like my patch for some other reason.  I think you're being
unreasonably inflexible.  As far as I can tell, you haven't tried it out
yet, or looked at the code, beyond a superficial glance at a large diff.

> If you want to do that, I'd rather you keep `lambda` as before and
> macro-expand it to a new form that has the new syntax.

There is no new syntax, as I keep telling you.

> Part of the cost of your change here is that it prevents future changes
> that might want to use a similar syntax for other purposes.
> IOW it "eats up namespace", tho "namespace" is not quite the right word.

I don't know what you're talking about here.

> >> - changing the representation of functions (both bytecode and interpreted).
> > Don't forget the native compiled, too!

> That's more internal and it's currently not used for closures, so I find
> this part of the cost negligible.

> > Only the change to interpreted code is non-trivial.  For the compiled
> > code, there has simply been an extra field inserted at or near the end
> > of the function structures.

> For bytecode it's still an extra field (which we have to pay for every
> closure we create) thus increasing memory use.  Plus the incompatible
> change of position of the interactive form (tho I find this less of an
> problem).

I think you're being thoroughly unrealistic here.  RAM in a modern
machine is measured in gigabytes, not kilobytes.  There are around 36k
DEFUNs/defuns/defmacros in Emacs.  Even if all were simultaneously
loaded into RAM, that would only be around a quarter of a megabyte of
storage used up by my patch.

If storage were really that critical, we would never have changed from
4-byte Lisp_Objects to 8-byte ones, something which will have hugely
increased the storage used.

The change in the interactive form in byte compiled functions isn't
incompatible with anything that uses the proper interfaces.

> Maybe in your tests, the increase is negligible, but remember: every
> user will pay this tiny cost, and most of the users will never reap any
> benefit from it, and those who do will only do very rarely.

Developers will benefit.

> It's a good change for us hard-core Emacs hackers, but it has to come
> *very* cheap because the gain is really low for other users.

It _is_ very cheap.

> >> Storing the info in the docstring would fix the second and third point.
> >> Using the FILE+LINE+COL info would fix the first.
> > I don't know what you mean by "docstring".

> Docstrings live in the `.elc` files.

OK, you've got your own vocabulary on this point.  To most people, a doc
string is something abstract, which is created in *scratch* or a .el
file, gets evaluated with, say C-x C-e, gets compiled by compile-defun,
and gets native compiled, too, all the while remaining that same doc
string.

> > You could mean either in the ..elc/.eln file or in the function
> > structure or both.  I don't see docstring storage as a very good idea.

> Probably because you think of it as a docstring only.
> Think of it as "out of line storage" where we can keep introspection
> data for which we don't want to pay *any* cost (other than disk space
> use) in the normal execution of code (i.e. when calling the closure or
> when creating the closure) such as:
> - documentation
> - human-readable arglist
> - source code location
> - debug info
> - a copy of the source code
> - ...

Yes, I think of the "docstring" as the doc string.  I don't think it a
good idea to cram other things into it which aren't the doc string.

> > It seems like a kludge, and would complicate Emacs's handling, much as
> > the existing new code complicates it.

> I see it as an opportunity to clean up this kludge.

Go ahead, then.  Then we can compare your version with mine.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).