Making Emacs Lisp easier to debug

Making Emacs Lisp easier to debug

[Alan Mackenzie]

Hello, Emacs.

Despite some recent improvements, and some which are currently being
worked on, there are still scenarios where Lisp is difficult to debug.
It is possible that we might be able to improve matters.  The two
scenarios I have in mind at the moment are:

(i) Edebugging macro expansions.  What happens at the moment is that the
arguments given to the macro, when given a suitable edebug spec, get
edebugged, but the code generated by the macro remains opaque.  I
envision an enhancement to Edebug such that the code generated by the
macro would get instrumented, and the appropriate bit in the macro
source would get displayed when stepping through the containing
function.

(ii) Edebugging font locking code.  It would be nice to be able to debug
font locking code which is being called from redisplay, as well as other
hooks which are also called from redisplay.  At the moment this isn't
possible.  To make it possible would require redisplay to become
reentrant, so that a frame or window in the inner redisplay call 
could be where debugging of the outer redisplay call happens.  I don't
know how practicable it would be to enhance redisplay for this (it is
certainly possible).  Has anybody ever looked into this before?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Making Emacs Lisp easier to debug

[Eli Zaretskii]

> Date: Fri, 10 Nov 2023 20:56:54 +0000
> From: Alan Mackenzie <acm@muc.de>
> 
> (ii) Edebugging font locking code.  It would be nice to be able to debug
> font locking code which is being called from redisplay, as well as other
> hooks which are also called from redisplay.  At the moment this isn't
> possible.  To make it possible would require redisplay to become
> reentrant, so that a frame or window in the inner redisplay call 
> could be where debugging of the outer redisplay call happens.  I don't
> know how practicable it would be to enhance redisplay for this (it is
> certainly possible).  Has anybody ever looked into this before?

Yes, see jit-lock-debug-mode.  (It's described in the ELisp manual.)

Re: Making Emacs Lisp easier to debug

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> (ii) Edebugging font locking code.  It would be nice to be able to debug
>> font locking code which is being called from redisplay, as well as other
>> hooks which are also called from redisplay.  At the moment this isn't
>> possible.  To make it possible would require redisplay to become
>> reentrant, so that a frame or window in the inner redisplay call 
>> could be where debugging of the outer redisplay call happens.  I don't
>> know how practicable it would be to enhance redisplay for this (it is
>> certainly possible).  Has anybody ever looked into this before?
>
> Yes, see jit-lock-debug-mode.  (It's described in the ELisp manual.)

Also, I found built-in M-x font-lock-debug-fontify helpful.
And third-party package https://github.com/Lindydancer/font-lock-studio
Finally, `backtrace-on-redisplay-error' is useful.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Making Emacs Lisp easier to debug

[Alan Mackenzie]

Hello, Eli.

On Sat, Nov 11, 2023 at 08:52:12 +0200, Eli Zaretskii wrote:
> > Date: Fri, 10 Nov 2023 20:56:54 +0000
> > From: Alan Mackenzie <acm@muc.de>

> > (ii) Edebugging font locking code.  It would be nice to be able to debug
> > font locking code which is being called from redisplay, as well as other
> > hooks which are also called from redisplay.  At the moment this isn't
> > possible.  To make it possible would require redisplay to become
> > reentrant, so that a frame or window in the inner redisplay call 
> > could be where debugging of the outer redisplay call happens.  I don't
> > know how practicable it would be to enhance redisplay for this (it is
> > certainly possible).  Has anybody ever looked into this before?

> Yes, see jit-lock-debug-mode.  (It's described in the ELisp manual.)

:-)  I really meant has anybody looked into making redisplay reentrant.

As for jit-lock-debug-mode, it doesn't work in my setup.  It just
disables font-lock totally.  I suspect its timer isn't triggering for
some reason.  I haven't actually tried it yet in emacs -Q.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Making Emacs Lisp easier to debug

[Eli Zaretskii]

> Date: Sat, 11 Nov 2023 11:04:01 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> :-)  I really meant has anybody looked into making redisplay reentrant.

Good luck with that!

(I also don't understand why you think this will help with font-lock,
nor even how it would work in general, should it be possible.
Re-entering redisplay in the middle of a redisplay cycle means that
the outer redisplay didn't finish preparing the glyph matrices, and
what do you want the inner redisplay to do in such a case?)

Re: Making Emacs Lisp easier to debug

[Alan Mackenzie]

Hello, Eli.

On Sat, Nov 11, 2023 at 13:10:44 +0200, Eli Zaretskii wrote:
> > Date: Sat, 11 Nov 2023 11:04:01 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > :-)  I really meant has anybody looked into making redisplay reentrant.

> Good luck with that!

> (I also don't understand why you think this will help with font-lock,
> nor even how it would work in general, should it be possible.

With font lock, or any other Lisp hook called from redisplay, it should
be possible, in a recursive-edit loop, to run edebug, displaying on a
different frame.  That different frame would be running in the inner
redisplay while the outer redisplay would be suspended.

> Re-entering redisplay in the middle of a redisplay cycle means that
> the outer redisplay didn't finish preparing the glyph matrices, and
> what do you want the inner redisplay to do in such a case?

Work with the glyph matrices belonging to the inner redisplay whilst the
outer one is suspended.

> )

As I say, it is not clear whether or not this is possible or
practicable.  If it were, we could enhance edebug such that a function
in a font lock pattern, or on, say, window-scroll-functions could be
edebugged by doing nothing more than instrumenting it with C-u C-M-x.
Thus Lisp called from redisplay would cease to be an awkward special
case as far as debugging is concerned.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Making Emacs Lisp easier to debug

[Eli Zaretskii]

> Date: Sat, 11 Nov 2023 12:10:33 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > (I also don't understand why you think this will help with font-lock,
> > nor even how it would work in general, should it be possible.
> 
> With font lock, or any other Lisp hook called from redisplay, it should
> be possible, in a recursive-edit loop, to run edebug, displaying on a
> different frame.  That different frame would be running in the inner
> redisplay while the outer redisplay would be suspended.

Why do you need an inner redisplay for that?

And what will that frame show, given that the outer redisplay is
halfway through fontifying the text? what do you expect to see there,
and why?

> > Re-entering redisplay in the middle of a redisplay cycle means that
> > the outer redisplay didn't finish preparing the glyph matrices, and
> > what do you want the inner redisplay to do in such a case?
> 
> Work with the glyph matrices belonging to the inner redisplay whilst the
> outer one is suspended.

But that will immediately get you into the same problem, since the
offending window will get redisplayed by the inner redisplay, and will
again cause Edebug, etc., ad nauseam.

> As I say, it is not clear whether or not this is possible or
> practicable.  If it were, we could enhance edebug such that a function
> in a font lock pattern, or on, say, window-scroll-functions could be
> edebugged by doing nothing more than instrumenting it with C-u C-M-x.
> Thus Lisp called from redisplay would cease to be an awkward special
> case as far as debugging is concerned.

I think we should start by having a clear idea of what should such an
"inner redisplay" show and how, before we are talking about
implementing it.  My impression from what you wrote is that the idea
is way too vague to discuss the details, and what you call "inner
redisplay" is not what you want at all.

Re: Making Emacs Lisp easier to debug

[Alan Mackenzie]

Hello, Eli.

On Sat, Nov 11, 2023 at 15:47:51 +0200, Eli Zaretskii wrote:
> > Date: Sat, 11 Nov 2023 12:10:33 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > (I also don't understand why you think this will help with font-lock,
> > > nor even how it would work in general, should it be possible.

> > With font lock, or any other Lisp hook called from redisplay, it should
> > be possible, in a recursive-edit loop, to run edebug, displaying on a
> > different frame.  That different frame would be running in the inner
> > redisplay while the outer redisplay would be suspended.

> Why do you need an inner redisplay for that?

Because the outer redisplay would be the thing being debugged, and hence
not in a position to display the progress of edebug.

> And what will that frame show, given that the outer redisplay is
> halfway through fontifying the text? what do you expect to see there,
> and why?

I think the scope of a redisplay operation could change from everything
to a single frame.  So in normal operation a high level routine would
call redisplay2 for each of Emacs's frames in turn.

While stepping through a hook in edebug in the inner redisplay, the
outer redisplay would (I think) carry on looking like it did before the
outer redisplay started.  Or, possibly, it might look like a bare frame,
I'm not sure.

> > > Re-entering redisplay in the middle of a redisplay cycle means that
> > > the outer redisplay didn't finish preparing the glyph matrices, and
> > > what do you want the inner redisplay to do in such a case?

> > Work with the glyph matrices belonging to the inner redisplay whilst the
> > outer one is suspended.

> But that will immediately get you into the same problem, since the
> offending window will get redisplayed by the inner redisplay, and will
> again cause Edebug, etc., ad nauseam.

We could put a limit on the nesting depth of redisplay nesting that
edebug would cope with, a small integer (2 or 3, probably).  That is
very similar to what edebug currently does during a redisplay, i.e. it
bypasses edebug's processing entirely.

> > As I say, it is not clear whether or not this is possible or
> > practicable.  If it were, we could enhance edebug such that a function
> > in a font lock pattern, or on, say, window-scroll-functions could be
> > edebugged by doing nothing more than instrumenting it with C-u C-M-x.
> > Thus Lisp called from redisplay would cease to be an awkward special
> > case as far as debugging is concerned.

> I think we should start by having a clear idea of what should such an
> "inner redisplay" show and how, before we are talking about
> implementing it.

I agree.

> My impression from what you wrote is that the idea is way too vague to
> discuss the details, ....

Yes.  It's currently just a vague idea, and it's not even clear as yet
whether it's practicable.  But if it is, I think it would be worthwhile
implementing.

> .... and what you call "inner redisplay" is not what you want at all.

I don't understand why you say this.  Do you see any particular problems
with it (in addition to what you've already written), or perhaps have a
better scheme for running edebug on redisplay's several Lisp hooks?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Making Emacs Lisp easier to debug

[Eli Zaretskii]

> Date: Sat, 11 Nov 2023 14:56:17 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > With font lock, or any other Lisp hook called from redisplay, it should
> > > be possible, in a recursive-edit loop, to run edebug, displaying on a
> > > different frame.  That different frame would be running in the inner
> > > redisplay while the outer redisplay would be suspended.
> 
> > Why do you need an inner redisplay for that?
> 
> Because the outer redisplay would be the thing being debugged, and hence
> not in a position to display the progress of edebug.

But you don't want to debug redisplay, you want to debug the font-lock
code called by redisplay.

> > And what will that frame show, given that the outer redisplay is
> > halfway through fontifying the text? what do you expect to see there,
> > and why?
> 
> I think the scope of a redisplay operation could change from everything
> to a single frame.

Not really, not with the current code design.

And you don't want a separate frame, you want a separate window.

> While stepping through a hook in edebug in the inner redisplay, the
> outer redisplay would (I think) carry on looking like it did before the
> outer redisplay started.  Or, possibly, it might look like a bare frame,
> I'm not sure.

So basically, the outer redisplay doesn't exist.  All you need is for
it to call the font-lock code.  Which once again brings me to the
question: why isn't jit-lock-debug-mode not what you want?

> > > > Re-entering redisplay in the middle of a redisplay cycle means that
> > > > the outer redisplay didn't finish preparing the glyph matrices, and
> > > > what do you want the inner redisplay to do in such a case?
> 
> > > Work with the glyph matrices belonging to the inner redisplay whilst the
> > > outer one is suspended.
> 
> > But that will immediately get you into the same problem, since the
> > offending window will get redisplayed by the inner redisplay, and will
> > again cause Edebug, etc., ad nauseam.
> 
> We could put a limit on the nesting depth of redisplay nesting that
> edebug would cope with, a small integer (2 or 3, probably).

That will prevent infinite recursion, but it won't solve your problem,
because you do want the "inner" redisplay to finish, so it shows the
window where you are running the debugger.

> > .... and what you call "inner redisplay" is not what you want at all.
> 
> I don't understand why you say this.

I'm saying that what you seem to want is an ability to display a
single window, the one where you run the debugger.  All the rest is of
no interest to you, and can, for example, be stopped in its tracks.
The solution to that is not a reentrant redisplay, it's something
else.

Re: Making Emacs Lisp easier to debug

[Alan Mackenzie]

Hello, Eli.

On Sat, Nov 11, 2023 at 18:01:08 +0200, Eli Zaretskii wrote:
> > Date: Sat, 11 Nov 2023 14:56:17 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > > With font lock, or any other Lisp hook called from redisplay, it should
> > > > be possible, in a recursive-edit loop, to run edebug, displaying on a
> > > > different frame.  That different frame would be running in the inner
> > > > redisplay while the outer redisplay would be suspended.

> > > Why do you need an inner redisplay for that?

> > Because the outer redisplay would be the thing being debugged, and hence
> > not in a position to display the progress of edebug.

> But you don't want to debug redisplay, you want to debug the font-lock
> code called by redisplay.

Or the code called by any other hook in redisplay, including, perhaps,
the jit-lock mechanism itself.

> > > And what will that frame show, given that the outer redisplay is
> > > halfway through fontifying the text? what do you expect to see there,
> > > and why?

> > I think the scope of a redisplay operation could change from everything
> > to a single frame.

> Not really, not with the current code design.

I see 14 FOR_EACH_FRAMEs in xdisp.c.  Are you saying that those loops
could not be rearranged to do everything in one frame first, followed by
doing everything in the next, and so on?  Surely each frame's redisplay
is independent of all others'.

> And you don't want a separate frame, you want a separate window.

Possibly.  A separate window feels like more complicated to implement,
even if not by much.

> > While stepping through a hook in edebug in the inner redisplay, the
> > outer redisplay would (I think) carry on looking like it did before the
> > outer redisplay started.  Or, possibly, it might look like a bare frame,
> > I'm not sure.

> So basically, the outer redisplay doesn't exist.  All you need is for
> it to call the font-lock code.  Which once again brings me to the
> question: why isn't jit-lock-debug-mode not what you want?

It's rather restricted: it won't help debug a window_scroll_functions
function, for example.  It won't help debug the jit-lock mechanism
itself, should it be decided to amend it.  Also, as I mentioned, it's not
working for me at the moment.  It gives the impression of being an
unfinished piece of code.

> > > > > Re-entering redisplay in the middle of a redisplay cycle means that
> > > > > the outer redisplay didn't finish preparing the glyph matrices, and
> > > > > what do you want the inner redisplay to do in such a case?

> > > > Work with the glyph matrices belonging to the inner redisplay whilst the
> > > > outer one is suspended.

> > > But that will immediately get you into the same problem, since the
> > > offending window will get redisplayed by the inner redisplay, and will
> > > again cause Edebug, etc., ad nauseam.

> > We could put a limit on the nesting depth of redisplay nesting that
> > edebug would cope with, a small integer (2 or 3, probably).

> That will prevent infinite recursion, but it won't solve your problem,
> because you do want the "inner" redisplay to finish, so it shows the
> window where you are running the debugger.

You're pressing me to defend a design I haven't even formulated yet.  ;-)
I envisage the inner redisplay simply running, including to a finish,
whilst the outer redisplay is suspended awaiting edebug to terminate.

> > > .... and what you call "inner redisplay" is not what you want at all.

> > I don't understand why you say this.

> I'm saying that what you seem to want is an ability to display a
> single window, the one where you run the debugger.  All the rest is of
> no interest to you, and can, for example, be stopped in its tracks.
> The solution to that is not a reentrant redisplay, it's something
> else.

But what?  While the debugger is debugging a redisplay hook (say,
window-scroll-functions) that invocation of redisplay is stationary.  The
frame or window where we're running the debugger requires a running
redisplay, however.  I can't see how we can achieve this without two
instances of redisplay, one stationary waiting for its hook to finish
being debugged, the other working for and displaying edebug's progress.
What am I not seeing, here?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Making Emacs Lisp easier to debug

[Eli Zaretskii]

> Date: Sat, 11 Nov 2023 17:23:53 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > Because the outer redisplay would be the thing being debugged, and hence
> > > not in a position to display the progress of edebug.
> 
> > But you don't want to debug redisplay, you want to debug the font-lock
> > code called by redisplay.
> 
> Or the code called by any other hook in redisplay, including, perhaps,
> the jit-lock mechanism itself.

That's not important, so let's please not introduce tangents.  This
subject is complicated enough as it is.

> > > I think the scope of a redisplay operation could change from everything
> > > to a single frame.
> 
> > Not really, not with the current code design.
> 
> I see 14 FOR_EACH_FRAMEs in xdisp.c.  Are you saying that those loops
> could not be rearranged to do everything in one frame first, followed by
> doing everything in the next, and so on?

I was saying that the current design is not per-frame.  If you look
closer, you will see that between the FOR_EACH_FRAMEs there's
non-trivial code, so rearranging them might not be as easy and obvious
as it might sound at first sight.

> Surely each frame's redisplay is independent of all others'.

Look closer: it is not, at least not as it is now.  For example, did
you look at propagate_buffer_redisplay, which is called outside any
FOR_EACH_FRAME loop?

> > And you don't want a separate frame, you want a separate window.
> 
> Possibly.  A separate window feels like more complicated to implement,
> even if not by much.

No, it is not.  The Emacs redisplay actually works by windows.

> > > While stepping through a hook in edebug in the inner redisplay, the
> > > outer redisplay would (I think) carry on looking like it did before the
> > > outer redisplay started.  Or, possibly, it might look like a bare frame,
> > > I'm not sure.
> 
> > So basically, the outer redisplay doesn't exist.  All you need is for
> > it to call the font-lock code.  Which once again brings me to the
> > question: why isn't jit-lock-debug-mode not what you want?
> 
> It's rather restricted: it won't help debug a window_scroll_functions
> function, for example.

So you want to debug window-scroll-functions, not font-lock?  Then why
did you start by talking about font-lock?

And if you are talking about debugging window-scroll-functions, what
prevents you from simply calling such a function interactively, and
debugging it then?  window-scroll-functions generally don't affect
redisplay (it is a very bad idea for them to try), so redisplay is not
involved here at all, it is only a tool for calling the hook.

> It won't help debug the jit-lock mechanism itself, should it be
> decided to amend it.

What do you mean by "jit-lock mechanism"?  Most of what I call
"jit-lock mechanism" is in C anyway.

> Also, as I mentioned, it's not working for me at the moment.  It
> gives the impression of being an unfinished piece of code.

Then how about fixing the problems there before talking about these
far-reaching ideas?  Surely, jit-lock-debug-mode is much closer to a
satisfactory solution than the ideas of "re-entering redisplay"?

> > > We could put a limit on the nesting depth of redisplay nesting that
> > > edebug would cope with, a small integer (2 or 3, probably).
> 
> > That will prevent infinite recursion, but it won't solve your problem,
> > because you do want the "inner" redisplay to finish, so it shows the
> > window where you are running the debugger.
> 
> You're pressing me to defend a design I haven't even formulated yet.  ;-)

No, I'm trying to convince you that you have no design to talk about ;-)

> I envisage the inner redisplay simply running, including to a finish,
> whilst the outer redisplay is suspended awaiting edebug to terminate.

Ask yourself why you need another redisplay to begin with.

> > > > .... and what you call "inner redisplay" is not what you want at all.
> 
> > > I don't understand why you say this.
> 
> > I'm saying that what you seem to want is an ability to display a
> > single window, the one where you run the debugger.  All the rest is of
> > no interest to you, and can, for example, be stopped in its tracks.
> > The solution to that is not a reentrant redisplay, it's something
> > else.
> 
> But what?  While the debugger is debugging a redisplay hook (say,
> window-scroll-functions) that invocation of redisplay is stationary.  The
> frame or window where we're running the debugger requires a running
> redisplay, however.  I can't see how we can achieve this without two
> instances of redisplay, one stationary waiting for its hook to finish
> being debugged, the other working for and displaying edebug's progress.
> What am I not seeing, here?

I'm saying that you want the ability to display just one window (well,
actually two: the window with the code being debugged and the
mini-window).  This has nothing to do with re-entering redisplay, it
has everything to do with a finer control on what redisplay does when
entered.

Re: Making Emacs Lisp easier to debug

[Alan Mackenzie]

Hello, Eli.

On Sat, Nov 11, 2023 at 19:54:23 +0200, Eli Zaretskii wrote:
> > Date: Sat, 11 Nov 2023 17:23:53 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > > Because the outer redisplay would be the thing being debugged, and hence
> > > > not in a position to display the progress of edebug.

> > > But you don't want to debug redisplay, you want to debug the font-lock
> > > code called by redisplay.

> > Or the code called by any other hook in redisplay, including, perhaps,
> > the jit-lock mechanism itself.

> That's not important, so let's please not introduce tangents.  This
> subject is complicated enough as it is.

When I was starting to use edebug, a long time ago, I would frequently
ask myself why, after instrumenting some code and running it, it would
not bring up an edebug window.  I'm still asking myself that question,
but now I'm attempting to provide some answers.

It's also worth pointing out that a little over a year ago, when I was
writing backtrace-on-redisplay-error, you applied a fair bit of pressure
to me to handle _all_ redisplay's hooks, not just the font lock ones.
Now, by contrast, you're saying all but part of the font locking stuff is
tangents and inessential.

[ .... ]

> > > And you don't want a separate frame, you want a separate window.

> > Possibly.  A separate window feels like more complicated to implement,
> > even if not by much.

> No, it is not.  The Emacs redisplay actually works by windows.

OK.

> > > > While stepping through a hook in edebug in the inner redisplay, the
> > > > outer redisplay would (I think) carry on looking like it did before the
> > > > outer redisplay started.  Or, possibly, it might look like a bare frame,
> > > > I'm not sure.

> > > So basically, the outer redisplay doesn't exist.  All you need is for
> > > it to call the font-lock code.  Which once again brings me to the
> > > question: why isn't jit-lock-debug-mode not what you want?

> > It's rather restricted: it won't help debug a window_scroll_functions
> > function, for example.

> So you want to debug window-scroll-functions, not font-lock?  Then why
> did you start by talking about font-lock?

Must you be so aggressive, Eli?  What I want is to be able to instrument
a function for edebug, run it, and be able to step through it, all the
time, not just most of the time.  I think I said that in my opening post
in this thread.

> And if you are talking about debugging window-scroll-functions, what
> prevents you from simply calling such a function interactively, and
> debugging it then?  window-scroll-functions generally don't affect
> redisplay (it is a very bad idea for them to try), so redisplay is not
> involved here at all, it is only a tool for calling the hook.

Nothing prevents people doing that.  Indeed they're forced to go through
such uncomfortable contrivances.  I don't think people enjoy having to do
so, though.  I certainly don't.  I'm trying to propose an improvement.

> > It won't help debug the jit-lock mechanism itself, should it be
> > decided to amend it.

> What do you mean by "jit-lock mechanism"?  Most of what I call
> "jit-lock mechanism" is in C anyway.

I mean the Lisp functions which are on fontification-functions.

> > Also, as I mentioned, it's not working for me at the moment.  It
> > gives the impression of being an unfinished piece of code.

> Then how about fixing the problems there before talking about these
> far-reaching ideas?  Surely, jit-lock-debug-mode is much closer to a
> satisfactory solution than the ideas of "re-entering redisplay"?

I don't think so, no.  The notion I'm playing with at the moment is that
inessential design elements in redisplay are preventing edebug working
smoothly, all the time.  Sure, there are ad-hoc workarounds for parts of
font locking, and some of the other redisplay hooks, but it would be
better to amend redisplay such that edebug simply worked.  That is what I
am trying to propose.

That said, it would indeed be worthwhile trying to fix
jit-lock-debug-mode.

[ .... ]

> > You're pressing me to defend a design I haven't even formulated yet.  ;-)

> No, I'm trying to convince you that you have no design to talk about ;-)

But it doesn't feel like you're trying to help me in the formulation of
such a design.

> > I envisage the inner redisplay simply running, including to a finish,
> > whilst the outer redisplay is suspended awaiting edebug to terminate.

> Ask yourself why you need another redisplay to begin with.

I think I've addressed this point several times.  The redisplay whose
hooks are being debugged in edebug will not be able to display that
edebug session, so we need another one.  Are you saying I'm missing
something, here?

> > > > > .... and what you call "inner redisplay" is not what you want at all.

> > > > I don't understand why you say this.

> > > I'm saying that what you seem to want is an ability to display a
> > > single window, the one where you run the debugger.  All the rest is of
> > > no interest to you, and can, for example, be stopped in its tracks.
> > > The solution to that is not a reentrant redisplay, it's something
> > > else.

> > But what?  While the debugger is debugging a redisplay hook (say,
> > window-scroll-functions) that invocation of redisplay is stationary.  The
> > frame or window where we're running the debugger requires a running
> > redisplay, however.  I can't see how we can achieve this without two
> > instances of redisplay, one stationary waiting for its hook to finish
> > being debugged, the other working for and displaying edebug's progress.
> > What am I not seeing, here?

> I'm saying that you want the ability to display just one window (well,
> actually two: the window with the code being debugged and the
> mini-window).  This has nothing to do with re-entering redisplay, it
> has everything to do with a finer control on what redisplay does when
> entered.

I don't feel that addresses my point.  What is this finer control you're
talking about, and how will it help me step through a redisplay hook with
edebug?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Making Emacs Lisp easier to debug

[Eli Zaretskii]

> Date: Sat, 11 Nov 2023 19:55:29 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> It's also worth pointing out that a little over a year ago, when I was
> writing backtrace-on-redisplay-error, you applied a fair bit of pressure
> to me to handle _all_ redisplay's hooks, not just the font lock ones.
> Now, by contrast, you're saying all but part of the font locking stuff is
> tangents and inessential.

That's not what I said.  I said that debugging the jit-lock mechanism
doesn't bump into the problem of being unable to Edebug font-lock
invoked by jit-lock.

IOW, I'm trying to keep this discussion focused on specific issues,
rather than let it degenerate into another futile dispute, by lumping
together unrelated issues and tangents.

> > So you want to debug window-scroll-functions, not font-lock?  Then why
> > did you start by talking about font-lock?
> 
> Must you be so aggressive, Eli?

It isn't aggression, it's frustration.  I have, as you might imagine,
very little time to waste on pointless discussions, so I get
frustrated when, after no less than 3 messages of me trying to help
you solve some problem you seem to be raising, it turns out you have
something very different in mind, which you didn't clearly state until
now.

Please try to describe the issue more clearly and comprehensively next
time, and save me and others from wasting efforts on looking up
solutions for problems you seem to raise, solutions you don't really
mean to use, because you are actually looking for something very
different.

> What I want is to be able to instrument a function for edebug, run
> it, and be able to step through it, all the time, not just most of
> the time.  I think I said that in my opening post in this thread.

You haven't, not in so many words.  You certainly didn't say that you
won't accept any solution except the one that allows you to "re-enter
redisplay".

In any case, my point is that "reentrant redisplay" is not the path
towards making Edebug capable of debugging Lisp called from C in
general and from redisplay in particular.

> > And if you are talking about debugging window-scroll-functions, what
> > prevents you from simply calling such a function interactively, and
> > debugging it then?  window-scroll-functions generally don't affect
> > redisplay (it is a very bad idea for them to try), so redisplay is not
> > involved here at all, it is only a tool for calling the hook.
> 
> Nothing prevents people doing that.  Indeed they're forced to go through
> such uncomfortable contrivances.  I don't think people enjoy having to do
> so, though.  I certainly don't.  I'm trying to propose an improvement.

I don't see what is "contrived" in invoking a function you want to
debug.

> > > It won't help debug the jit-lock mechanism itself, should it be
> > > decided to amend it.
> 
> > What do you mean by "jit-lock mechanism"?  Most of what I call
> > "jit-lock mechanism" is in C anyway.
> 
> I mean the Lisp functions which are on fontification-functions.

And jit-lock-debug-mode doesn't use that code?  Which code
specifically?

> > > Also, as I mentioned, it's not working for me at the moment.  It
> > > gives the impression of being an unfinished piece of code.
> 
> > Then how about fixing the problems there before talking about these
> > far-reaching ideas?  Surely, jit-lock-debug-mode is much closer to a
> > satisfactory solution than the ideas of "re-entering redisplay"?
> 
> I don't think so, no.  The notion I'm playing with at the moment is that
> inessential design elements in redisplay are preventing edebug working
> smoothly, all the time.  Sure, there are ad-hoc workarounds for parts of
> font locking, and some of the other redisplay hooks, but it would be
> better to amend redisplay such that edebug simply worked.  That is what I
> am trying to propose.

So you decided, up front, that your goal is to make redisplay
reentrant, because you are of the opinion that its being non-reentrant
is due to "inessential design elements", and therefore you are going
to reject any other practical proposal for debugging font-lock or any
Lisp invoked from redisplay because they don't involve removing what
you perceive as "inessential design elements" from redisplay?

Do you see now why I feel frustration in this kind of discussion?  If
you were to explain the above intention and motivation from the
get-go, I would probably have decided not to respond to your message
at all.

> > > You're pressing me to defend a design I haven't even formulated yet.  ;-)
> 
> > No, I'm trying to convince you that you have no design to talk about ;-)
> 
> But it doesn't feel like you're trying to help me in the formulation of
> such a design.

Because you don't listen to what I'm saying.  You instead are trying
to prove that I'm wrong and don't know what I'm talking about.

> > Ask yourself why you need another redisplay to begin with.
> 
> I think I've addressed this point several times.  The redisplay whose
> hooks are being debugged in edebug will not be able to display that
> edebug session, so we need another one.  Are you saying I'm missing
> something, here?

Yes, you are missing that the Edebug session is shown in a separate
window, not in the same window whose display triggered the code being
debugged.

> > I'm saying that you want the ability to display just one window (well,
> > actually two: the window with the code being debugged and the
> > mini-window).  This has nothing to do with re-entering redisplay, it
> > has everything to do with a finer control on what redisplay does when
> > entered.
> 
> I don't feel that addresses my point.  What is this finer control you're
> talking about, and how will it help me step through a redisplay hook with
> edebug?

The finer control I allude to is the ability to display specific
windows.  That's not what redisplay currently does, see
redisplay_windows (which is called from a FOR_EACH_FRAME loop).

Re: Making Emacs Lisp easier to debug

[Alan Mackenzie]

Hello, Eli.

On Sun, Nov 12, 2023 at 09:17:14 +0200, Eli Zaretskii wrote:
> > Date: Sat, 11 Nov 2023 19:55:29 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

[ .... ]

> > Must you be so aggressive, Eli?

> It isn't aggression, it's frustration.  I have, as you might imagine,
> very little time to waste on pointless discussions, so I get
> frustrated when, after no less than 3 messages of me trying to help
> you solve some problem you seem to be raising, it turns out you have
> something very different in mind, which you didn't clearly state until
> now.

> Please try to describe the issue more clearly and comprehensively next
> time, and save me and others from wasting efforts on looking up
> solutions for problems you seem to raise, solutions you don't really
> mean to use, because you are actually looking for something very
> different.

In this thread, we have not been communicating effectively.  I have
clearly failed to get my idea across to you, and on my side, I have not
been able to relate what you have said to what I tried to say.

I'm sorry about this.

I will examine the thread again, and try to work out how I could have
done better, in the hope that next time it _will_ be better.

I can always try to implement my idea, and then it will only be my own
time.  But I've got plenty of other things to do at the moment, so this
won't be happening soon.

[ .... ]

> > > > Also, as I mentioned, it[jit-lock-debug-mode]'s not working for
> > > > me at the moment.  It gives the impression of being an unfinished
> > > > piece of code.

> > > Then how about fixing the problems there before talking about these
> > > far-reaching ideas?  Surely, jit-lock-debug-mode is much closer to a
> > > satisfactory solution than the ideas of "re-entering redisplay"?

I spent some time on jit-lock-debug-mode last night, and bug #67116
"byte-comp-let: reversing the order of evaluation of the clauses CAN make
a difference." resulted from that.  I managed to get jit-lock-debug-mode
working by replacing a `let' by a `let*'.

So there will probably be a fix for jit-lock-debug-mode committed soon,
whether from Stefan or from me.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Making Emacs Lisp easier to debug

[Eli Zaretskii]

> Date: Sun, 12 Nov 2023 12:08:48 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> I will examine the thread again, and try to work out how I could have
> done better, in the hope that next time it _will_ be better.

Thank you.

XY Problems (tangent, related to common discussion issue here)

[chad]

[-- Attachment #1: Type: text/plain, Size: 3798 bytes --]

This exchange between two incredibly seasoned, experienced contributors to
emacs provide a near-textbook example of what another contributor recently
brought up, "an XY Problem". Eli's recent message highlights the issue
pretty clearly;

On Sun, Nov 12, 2023 at 2:18 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > Now, by contrast, you're saying all but part of the font locking stuff is
> > tangents and inessential.
>
> That's not what I said.  I said that debugging the jit-lock mechanism
> doesn't bump into the problem of being unable to Edebug font-lock
> invoked by jit-lock.
>
> IOW, I'm trying to keep this discussion focused on specific issues,
> rather than let it degenerate into another futile dispute, by lumping
> together unrelated issues and tangents.
>
> > > So you want to debug window-scroll-functions, not font-lock?  Then why
> > > did you start by talking about font-lock?
> >
> > Must you be so aggressive, Eli?
>
> It isn't aggression, it's frustration.  I have, as you might imagine,
> very little time to waste on pointless discussions, so I get
> frustrated when, after no less than 3 messages of me trying to help
> you solve some problem you seem to be raising, it turns out you have
> something very different in mind, which you didn't clearly state until
> now.
>
> Please try to describe the issue more clearly and comprehensively next
> time, and save me and others from wasting efforts on looking up
> solutions for problems you seem to raise, solutions you don't really
> mean to use, because you are actually looking for something very
> different.
>

While I hadn't heard the term "XY problem" until João referenced it
recently, I've seen the problem itself many times. At heart, the issue
comes out of some variation of:

* a user encounters an actual, concrete problem in practice
* the user diagnoses the problem as the result of some cause
* the user formulates a plan/strategy/etc to resolve (or at least
thoroughly investigate) the diagnosed cause (critically, *not* the original
problem)
* the user appeals to external resources for help *with their attempt to
resolve/investigate the diagnosed cause.

From what I've observed, this problem is significantly more likely to occur
when the people involved are higher skilled, more expert, and more
knowledgeable, both in general and in/near the fields in question. (I
myself saw it happen multiple times a week over several years of working
with the various technical support groups for MIT's campus computing
systems, many years back; I also saw it happen frequently in more
widespread on-line help systems (Usenet, internet fora, mailing lists), and
I think we all can agree that it's happened on emacs-devel some non-trivial
number of times).

In my experience, the *most* common response from the hypothetical original
user above after being told that their original diagnosis was mistaken is
refusal to believe. (There is apparently a bunch of psychology that helps
explain why, but little dispute around whether it happens or not.) The most
effective methods we found for dealing with it were A.) develop the
expectation that the first steps in providing technical support is /always/
to start as close as possible to "what are you trying to do?", and B.)
encourage everyone involved to remember that this is fundamentally an issue
of communications, not skill, knowledge, acumen, etc.

I want to be clear that I'm not trying to call out anyone in particular
with this message; this is a pernicious pattern that is very common, and
there are well-worn psychological and cognitive science principles that
make it much easier to fall into this trap than to avoid it.

Thanks for reading this far; I hope that it helps,
~Chad

[-- Attachment #2: Type: text/html, Size: 4424 bytes --]