bug#61847: debug-early-backtrace only works some of the time.

bug#61847: debug-early-backtrace only works some of the time.

[Alan Mackenzie]

Hello, Emacs.

In the master branch:

Sometimes, instead of outputting a backtrace, debug-early throws an
error, like the second line of:

Error: cl-assertion-failed ((stringp typename))
Symbol's function definition is void: cl-defgeneric

..  To reproduce this failure, apply the following patch to src/eval.c,
then attempt a make bootstrap:

diff --git a/src/eval.c b/src/eval.c
index d42f7ffe894..8a88f5894b1 100644
--- a/src/eval.c
+++ b/src/eval.c
@@ -542,13 +542,13 @@ DEFUN ("function", Ffunction, Sfunction, 1, UNEVALLED, 0,
 	  && (EQ (QCdocumentation, XCAR (tmp))))
 	{ /* Handle the special (:documentation <form>) to build the docstring
 	     dynamically.  */
-	  Lisp_Object docstring = eval_sub (Fcar (XCDR (tmp)));
-	  if (SYMBOLP (docstring) && !NILP (docstring))
-	    /* Hack for OClosures: Allow the docstring to be a symbol
-             * (the OClosure's type).  */
-	    docstring = Fsymbol_name (docstring);
-	  CHECK_STRING (docstring);
-	  cdr = Fcons (XCAR (cdr), Fcons (docstring, XCDR (XCDR (cdr))));
+	  /* Lisp_Object docstring = eval_sub (Fcar (XCDR (tmp))); */
+	  /* if (SYMBOLP (docstring) && !NILP (docstring)) */
+	  /*   /\* Hack for OClosures: Allow the docstring to be a symbol */
+          /*    * (the OClosure's type).  *\/ */
+	  /*   docstring = Fsymbol_name (docstring); */
+	  /* CHECK_STRING (docstring); */
+	  /* cdr = Fcons (XCAR (cdr), Fcons (docstring, XCDR (XCDR (cdr)))); */
 	}
       if (NILP (Vinternal_make_interpreted_closure_function))
         return Fcons (Qclosure, Fcons (Vinternal_interpreter_environment, cdr));

..

The cause of the problem was patch

commit 08108a856a544a80d11b1e9e437fe6c45e25adec
Author: Stefan Monnier <monnier@iro.umontreal.ca>
Date:   Fri Apr 29 22:18:09 2022 -0400

    debug-early: Print bytecode in a more manageable way

    * lisp/emacs-lisp/debug-early.el (debug-early-backtrace):
    Escape newlines to and bytecodes to make backtraces slightly more
    readable.  Use `cl-prin1` when available.

, which made debug-early.el dependent on arbitrarily nested Lisp code, in
violation of its explicitly stated design goal to have _no_ dependence on
Lisp code.  Some of this Lisp simply fails to load.  It's not clear why
the patch was applied.

I propose fixing the bug by restoring the code to having no such
dependencies with the following patch:


diff --git a/lisp/emacs-lisp/debug-early.el b/lisp/emacs-lisp/debug-early.el
index 395498f2206..723269f3ea0 100644
--- a/lisp/emacs-lisp/debug-early.el
+++ b/lisp/emacs-lisp/debug-early.el
@@ -44,28 +44,21 @@ 'debug-early-backtrace
       (princ "\n")
       (let ((print-escape-newlines t)
             (print-escape-control-characters t)
-            (print-escape-nonascii t)
-            (prin1 (if (and (fboundp 'cl-prin1)
-                            ;; If we're being called while
-                            ;; bootstrapping, we won't be able to load
-                            ;; cl-print.
-                            (require 'cl-print nil t))
-                       #'cl-prin1
-                     #'prin1)))
+            (print-escape-nonascii t))
         (mapbacktrace
          #'(lambda (evald func args _flags)
              (let ((args args))
 	       (if evald
 	           (progn
 	             (princ "  ")
-	             (funcall prin1 func)
+	             (prin1 func)
 	             (princ "("))
 	         (progn
 	           (princ "  (")
 	           (setq args (cons func args))))
 	       (if args
 	           (while (progn
-	                    (funcall prin1 (car args))
+	                    (prin1 (car args))
 	                    (setq args (cdr args)))
 	             (princ " ")))
 	       (princ ")\n")))))))

..  Any objections to applying this patch?

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#61847: debug-early-backtrace only works some of the time.

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> The cause of the problem was patch
>
> commit 08108a856a544a80d11b1e9e437fe6c45e25adec
> Author: Stefan Monnier <monnier@iro.umontreal.ca>
> Date:   Fri Apr 29 22:18:09 2022 -0400
>
>     debug-early: Print bytecode in a more manageable way
>
>     * lisp/emacs-lisp/debug-early.el (debug-early-backtrace):
>     Escape newlines to and bytecodes to make backtraces slightly more
>     readable.  Use `cl-prin1` when available.
>
> , which made debug-early.el dependent on arbitrarily nested Lisp code, in
> violation of its explicitly stated design goal to have _no_ dependence on
> Lisp code.  Some of this Lisp simply fails to load.

Indeed (fboundp 'cl-prin1) was too optimistic a test since it just
checks whether we're before loading `loaddef.el` or after it, but
`cl-prin1` uses more (preloaded) features such as those provided by
`cl-generic` which is only loaded later.

> It's not clear why the patch was applied.

To get more readable backtraces when used after bootstrapping (as well as
in the later phases of bootstrapping).

> I propose fixing the bug by restoring the code to having no such
> dependencies with the following patch:

How 'bout the patch below instead.
Maybe we should instead try and check whether we're after the bootstrap
(not sure what would be the corresponding test).


        Stefan


diff --git a/lisp/emacs-lisp/debug-early.el b/lisp/emacs-lisp/debug-early.el
index 395498f2206..65770a9c1d9 100644
--- a/lisp/emacs-lisp/debug-early.el
+++ b/lisp/emacs-lisp/debug-early.el
@@ -46,10 +46,13 @@ 'debug-early-backtrace
             (print-escape-control-characters t)
             (print-escape-nonascii t)
             (prin1 (if (and (fboundp 'cl-prin1)
+                            (fboundp 'cl-defmethod)
                             ;; If we're being called while
                             ;; bootstrapping, we won't be able to load
                             ;; cl-print.
-                            (require 'cl-print nil t))
+                            (condition-case nil
+                                (require 'cl-print nil t)
+                              (error nil)))
                        #'cl-prin1
                      #'prin1)))
         (mapbacktrace

bug#61847: debug-early-backtrace only works some of the time.

[Alan Mackenzie]

Hello, Stefan.

On Mon, Feb 27, 2023 at 14:15:22 -0500, Stefan Monnier wrote:
> > The cause of the problem was patch

> > commit 08108a856a544a80d11b1e9e437fe6c45e25adec
> > Author: Stefan Monnier <monnier@iro.umontreal.ca>
> > Date:   Fri Apr 29 22:18:09 2022 -0400

> >     debug-early: Print bytecode in a more manageable way

> >     * lisp/emacs-lisp/debug-early.el (debug-early-backtrace):
> >     Escape newlines to and bytecodes to make backtraces slightly more
> >     readable.  Use `cl-prin1` when available.

> > , which made debug-early.el dependent on arbitrarily nested Lisp code, in
> > violation of its explicitly stated design goal to have _no_ dependence on
> > Lisp code.  Some of this Lisp simply fails to load.

> Indeed (fboundp 'cl-prin1) was too optimistic a test since it just
> checks whether we're before loading `loaddef.el` or after it, but
> `cl-prin1` uses more (preloaded) features such as those provided by
> `cl-generic` which is only loaded later.

It violates the design goal of having no Lisp dependencies.  It was those
dependencies which broke the rock solid dependability of the original
code.

> > It's not clear why the patch was applied.

> To get more readable backtraces when used after bootstrapping (as well as
> in the later phases of bootstrapping).

More readable?  Just how is a backtrace produced using cl-prin1 more
readable than one using prin1?  They both look pretty much the same.  But
cl-prin1 only outputs partial information for some things, such as
compiled functions, so it is not a good choice.  debug-early-backtrace
should produce _complete_ backtraces.

> > I propose fixing the bug by restoring the code to having no such
> > dependencies with the following patch:

> How 'bout the patch below instead.
> Maybe we should instead try and check whether we're after the bootstrap
> (not sure what would be the corresponding test).

It might work.  It might work now, and fail in future releases of Emacs
should the loading mechanism get changed.  The original code using prin1
was rock solid, by design.

What if there was some bug in the loading mechanism, or a bug in
cl-print.el which prevented it loading cleanly, yet without triggering
the condition-case you suggest below?

Again, what's so readable about cl-prin1's output that makes it worth the
compromise in debug-early-backtrace's design?

And how will the contition-case you suggest help?  (require 'cl-print nil
t) returns non-nil in the pertinent circumstances.  Putting a
condition-case around that isn't going to change this.  Have you actually
tried out your patch?  What happened when you did?

>         Stefan


> diff --git a/lisp/emacs-lisp/debug-early.el b/lisp/emacs-lisp/debug-early.el
> index 395498f2206..65770a9c1d9 100644
> --- a/lisp/emacs-lisp/debug-early.el
> +++ b/lisp/emacs-lisp/debug-early.el
> @@ -46,10 +46,13 @@ 'debug-early-backtrace
>              (print-escape-control-characters t)
>              (print-escape-nonascii t)
>              (prin1 (if (and (fboundp 'cl-prin1)
> +                            (fboundp 'cl-defmethod)
>                              ;; If we're being called while
>                              ;; bootstrapping, we won't be able to load
>                              ;; cl-print.
> -                            (require 'cl-print nil t))
> +                            (condition-case nil
> +                                (require 'cl-print nil t)
> +                              (error nil)))
>                         #'cl-prin1
>                       #'prin1)))
>          (mapbacktrace

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#61847: debug-early-backtrace only works some of the time.

[Eli Zaretskii]

> Cc: 61847@debbugs.gnu.org
> Date: Tue, 28 Feb 2023 09:12:08 +0000
> From: Alan Mackenzie <acm@muc.de>
> 
> > How 'bout the patch below instead.
> > Maybe we should instead try and check whether we're after the bootstrap
> > (not sure what would be the corresponding test).
> 
> It might work.  It might work now, and fail in future releases of Emacs
> should the loading mechanism get changed.  The original code using prin1
> was rock solid, by design.

It isn't clear to me why you consider code that uses prin1 to be "rock
solid by design".  Please elaborate.

bug#61847: debug-early-backtrace only works some of the time.

[Alan Mackenzie]

Hello, Eli.

On Tue, Feb 28, 2023 at 14:16:40 +0200, Eli Zaretskii wrote:
> > Cc: 61847@debbugs.gnu.org
> > Date: Tue, 28 Feb 2023 09:12:08 +0000
> > From: Alan Mackenzie <acm@muc.de>

> > > How 'bout the patch below instead.
> > > Maybe we should instead try and check whether we're after the bootstrap
> > > (not sure what would be the corresponding test).

> > It might work.  It might work now, and fail in future releases of Emacs
> > should the loading mechanism get changed.  The original code using prin1
> > was rock solid, by design.

> It isn't clear to me why you consider code that uses prin1 to be "rock
> solid by design".  Please elaborate.

I meant that the specific piece of code which was the original code
(which happened to use prin1) was rock solid, not any other use of prin1.

The original code didn't depend on any other lisp being loaded.  Given
how much can go wrong with loading Lisp in the early bootstrap in a
version of Emacs being debugged, and which did go wrong in the bug
scenario, I say we're better off not trying to use cl-prin1 at this
stage.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#61847: debug-early-backtrace only works some of the time.

[Eli Zaretskii]

> Date: Tue, 28 Feb 2023 13:16:42 +0000
> Cc: monnier@iro.umontreal.ca, 61847@debbugs.gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > It isn't clear to me why you consider code that uses prin1 to be "rock
> > solid by design".  Please elaborate.
> 
> I meant that the specific piece of code which was the original code
> (which happened to use prin1) was rock solid, not any other use of prin1.
> 
> The original code didn't depend on any other lisp being loaded.  Given
> how much can go wrong with loading Lisp in the early bootstrap in a
> version of Emacs being debugged, and which did go wrong in the bug
> scenario, I say we're better off not trying to use cl-prin1 at this
> stage.

If you think that the original code is solid because no Lisp is
loaded, then all it takes to break that is that someone rewrites prin1
in Lisp.  Which happened with quite a few primitives in recent years,
and so it can happen with prin1 as well.  And if that does happen with
prin1, who will remember that this particular piece of code cannot
stand loading Lisp and will subtly break if that happens?

So if this feature needs some precautions when loading Lisp, we had
better introduced those precautions now, and tested them with Stefan's
code which uses cl-prin1 to make sure it works.  That way we will have
a more future-proof feature.

bug#61847: debug-early-backtrace only works some of the time.

[Alan Mackenzie]

Hello, Eli.

On Tue, Feb 28, 2023 at 16:22:30 +0200, Eli Zaretskii wrote:
> > Date: Tue, 28 Feb 2023 13:16:42 +0000
> > Cc: monnier@iro.umontreal.ca, 61847@debbugs.gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > It isn't clear to me why you consider code that uses prin1 to be "rock
> > > solid by design".  Please elaborate.

> > I meant that the specific piece of code which was the original code
> > (which happened to use prin1) was rock solid, not any other use of prin1.

> > The original code didn't depend on any other lisp being loaded.  Given
> > how much can go wrong with loading Lisp in the early bootstrap in a
> > version of Emacs being debugged, and which did go wrong in the bug
> > scenario, I say we're better off not trying to use cl-prin1 at this
> > stage.

> If you think that the original code is solid because no Lisp is
> loaded, then all it takes to break that is that someone rewrites prin1
> in Lisp.

That seems unlikely, since the C code needs some way of outputting
information before the Lisp even exists.

> Which happened with quite a few primitives in recent years, and so it
> can happen with prin1 as well.  And if that does happen with prin1,
> who will remember that this particular piece of code cannot stand
> loading Lisp and will subtly break if that happens?

I will remember it.  It would break, not at all subtly.  But you seem to
be arguing that because something can't be 100% perfect, it shouldn't be
improved.

> So if this feature needs some precautions when loading Lisp, we had
> better introduced those precautions now, and tested them with Stefan's
> code which uses cl-prin1 to make sure it works.

Why?  The code doesn't need cl-prin1, that's just a source of errors,
such as the one that happened.  I've asked Stefan why cl-prin1's output
is somehow more readable than straight prin1.

> That way we will have a more future-proof feature.

Not at all.  The original debug-early was as future-proof as it's
possible to get, and that was by deliberate design.  The current buggy
version, because of all the complexities of loading Lisp, is much less
dependable.  Even if some workaround is found, it will still be less
dependable.

Why do we want to use cl-prin1 here at all?  It doesn't appear to have
any advantages to offset its lack of dependability in early bootstrap.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#61847: debug-early-backtrace only works some of the time.

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> To get more readable backtraces when used after bootstrapping (as well as
>> in the later phases of bootstrapping).
>
> More readable?  Just how is a backtrace produced using cl-prin1 more
> readable than one using prin1?  They both look pretty much the same.  But
> cl-prin1 only outputs partial information for some things, such as
> compiled functions, so it is not a good choice.  debug-early-backtrace
> should produce _complete_ backtraces.

The output for compiled functions is the main one which I think is more
readable (among those that occur often in backtraces), so let's just
agree to disagree on this one.

> And how will the contition-case you suggest help?  (require 'cl-print nil
> t) returns non-nil in the pertinent circumstances.

The `noerror` argument of `require` doesn't silence the errors that
happen while loading the file, instead it prevents signaling an error
when the file is not found.

> Putting a condition-case around that isn't going to change this.

I'm hoping it will.

> Have you actually tried out your patch?

No.


        Stefan

bug#61847: debug-early-backtrace only works some of the time.

[Alan Mackenzie]

Hello, Stefan.

On Tue, Feb 28, 2023 at 12:36:32 -0500, Stefan Monnier wrote:
> >> To get more readable backtraces when used after bootstrapping (as well as
> >> in the later phases of bootstrapping).

> > More readable?  Just how is a backtrace produced using cl-prin1 more
> > readable than one using prin1?  They both look pretty much the same.  But
> > cl-prin1 only outputs partial information for some things, such as
> > compiled functions, so it is not a good choice.  debug-early-backtrace
> > should produce _complete_ backtraces.

> The output for compiled functions is the main one which I think is more
> readable (among those that occur often in backtraces), so let's just
> agree to disagree on this one.

No.  The point is too important not to resolve.

I think you're objectively wrong here.  The purpose of a backtrace is
not to enter a beauty contest.  Rather it's to provide the programmer
with as much information as reasonably possible to solve a bug.

The lack of output for compiled functions with cl-prin1 condemns it.  All
that appears is "#f(compiled-function)" together with an empty pair of
parentheses and a meaningless hex address.  What use is any of that in
debugging a batch mode bug?

prin1 by contrast prints the actual contents of the function - its byte
code string and its constant vector, among other things.  It may not be
as "readable", but it is infinitely more useful to the person trying to
debug a bug.

> > And how will the contition-case you suggest help?  (require 'cl-print nil
> > t) returns non-nil in the pertinent circumstances.

> The `noerror` argument of `require` doesn't silence the errors that
> happen while loading the file, instead it prevents signaling an error
> when the file is not found.

Whether that error is silenced or not is wholly unimportant.  The only
important thing here is to get a backtrace, and your patch will not help
do that.  Mine does help - I've tested it.

> > Putting a condition-case around that isn't going to change this.

> I'm hoping it will.

How can it possibly help get that backtrace?

> > Have you actually tried out your patch?

> No.

Please do so, and report.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#61847: debug-early-backtrace only works some of the time.

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> The output for compiled functions is the main one which I think is more
>> readable (among those that occur often in backtraces), so let's just
>> agree to disagree on this one.
>
> No.  The point is too important not to resolve.
>
> I think you're objectively wrong here.  The purpose of a backtrace is
> not to enter a beauty contest.  Rather it's to provide the programmer
> with as much information as reasonably possible to solve a bug.
>
> The lack of output for compiled functions with cl-prin1 condemns it.  All
> that appears is "#f(compiled-function)" together with an empty pair of
> parentheses and a meaningless hex address.  What use is any of that in
> debugging a batch mode bug?
>
> prin1 by contrast prints the actual contents of the function - its byte
> code string and its constant vector, among other things.  It may not be
> as "readable", but it is infinitely more useful to the person trying to
> debug a bug.

I'm not sure what you're expecting from me.  Obviously, I'm aware of
what you describe.  I just don't reach the same conclusion.

>> > And how will the contition-case you suggest help?  (require 'cl-print nil
>> > t) returns non-nil in the pertinent circumstances.
>> The `noerror` argument of `require` doesn't silence the errors that
>> happen while loading the file, instead it prevents signaling an error
>> when the file is not found.
> Whether that error is silenced or not is wholly unimportant.  The only
> important thing here is to get a backtrace,

The silencing of the error should help to get a backtrace since it
should let the code fall back to using `prin1`.


        Stefan

bug#61847: debug-early-backtrace only works some of the time.

[Eli Zaretskii]

> Cc: 61847@debbugs.gnu.org
> Date: Tue, 28 Feb 2023 19:52:33 +0000
> From: Alan Mackenzie <acm@muc.de>
> 
> The lack of output for compiled functions with cl-prin1 condemns it.  All
> that appears is "#f(compiled-function)" together with an empty pair of
> parentheses and a meaningless hex address.  What use is any of that in
> debugging a batch mode bug?

And what use is the meaningless stream of raw bytes that prin1
produces?

> prin1 by contrast prints the actual contents of the function - its byte
> code string and its constant vector, among other things.

And with some of the bytes interpreted by the terminal, it is _really_
useful.  To say nothing of attempting to post it in a bug report,
where it can ruin the entire email message.

I think you should re-evaluate your preferences, and base them on real
advantages and disadvantages, not on imaginary ones.  If we want our
backtraces to be more informative, not less, we should move farther
away of "dumb" output functions that just spill the guts and towards
more humanly-readable formatted description of the called functions.
IOW, make cl-prin1 smarter and teach it doing _more_ and do it
_better_, not less and worse.  For example, how about a more detailed,
but human-readable description of bytecode?

bug#61847: debug-early-backtrace only works some of the time.

[Eli Zaretskii]

> Cc: 61847@debbugs.gnu.org
> Date: Tue, 28 Feb 2023 19:52:33 +0000
> From: Alan Mackenzie <acm@muc.de>
> 
> The purpose of a backtrace is not to enter a beauty contest.  Rather
> it's to provide the programmer with as much information as
> reasonably possible to solve a bug.

Information that is humanly-readable and understandable, yes.  Not
just any information.  Showing raw bytes of the bytecode is not very
useful information, IMNSHO.  Or at least we could make it much more
useful, if we really want that part to be presented to the programmer.

> prin1 by contrast prints the actual contents of the function - its byte
> code string and its constant vector, among other things.  It may not be
> as "readable", but it is infinitely more useful to the person trying to
> debug a bug.

1 is "infinitely more" than zero, but it is still just 1.  Not 1`00
and not 1000.

IOW, just because relatively you get an "infinite" improvement, the
net improvement is still very small, and there's no reason to stop
there.

So your argument against cl-prin1, if taken to its logical conclusion,
should be rephrased as "let's improve cl-prin1", not "let's go back to
the infinitely useless prin1".

bug#61847: debug-early-backtrace only works some of the time.

[Alan Mackenzie]

Hello, Eli.

On Wed, Mar 01, 2023 at 14:16:11 +0200, Eli Zaretskii wrote:
> > Cc: 61847@debbugs.gnu.org
> > Date: Tue, 28 Feb 2023 19:52:33 +0000
> > From: Alan Mackenzie <acm@muc.de>

> > The lack of output for compiled functions with cl-prin1 condemns it.  All
> > that appears is "#f(compiled-function)" together with an empty pair of
> > parentheses and a meaningless hex address.  What use is any of that in
> > debugging a batch mode bug?

> And what use is the meaningless stream of raw bytes that prin1
> produces?

If that were indeed what was produced, none at all.  Because
debug-early-backtrace binds print-escape-control-characters to t, what
actually gets produced is a mixture of ascii characters and octal
escaped representations, just like you see when an .elc file is visited
in Emacs.

> > prin1 by contrast prints the actual contents of the function - its byte
> > code string and its constant vector, among other things.

> And with some of the bytes interpreted by the terminal, it is _really_
> useful.  To say nothing of attempting to post it in a bug report,
> where it can ruin the entire email message.

See above.

> I think you should re-evaluate your preferences, and base them on real
> advantages and disadvantages, not on imaginary ones.

What makes you think I'm not doing this already?  I wrote debug-early
last year because I _needed_ it.  I've a lot of experience using it, and
the way it prints (or rather used to print) a compiled function is/was
useful.  You can see how many arguments it takes.  You can see the byte
code string, hence enabling you to compare it visually with the contents
of a .elc file, should such already exist.  You can see the constant
vector.  You get an idea of how big the function is.  All these things
are helpful when you want to find out which particular lambda form is in
the backtrace.  Again, I'm talking from experience, not imagined
benefits.

> If we want our backtraces to be more informative, not less, we should
> move farther away of "dumb" output functions that just spill the guts
> and towards more humanly-readable formatted description of the called
> functions.

A wholesome long term goal I fully agree with.  For the here and now,
outputting "#f(compiled-function () <random hex address>" and nothing
else is not a step in that direction.

> IOW, make cl-prin1 smarter and teach it doing _more_ and do it
> _better_, not less and worse.  For example, how about a more detailed,
> but human-readable description of bytecode?

You mean a disassembly?  That's an idea, but it would be very bulky in a
backtrace, and hinder the visual comparison with the putatively same
byte code in a .elc buffer in Emacs.  A backtrace I got the other
evening was already 126 MB big.

Besides, the idea of debug-early is to generate a backtrace before all
the fancy Lisp facilities are available.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#61847: debug-early-backtrace only works some of the time.

[Eli Zaretskii]

> Date: Wed, 1 Mar 2023 15:22:29 +0000
> Cc: monnier@iro.umontreal.ca, 61847@debbugs.gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > And what use is the meaningless stream of raw bytes that prin1
> > produces?
> 
> If that were indeed what was produced, none at all.  Because
> debug-early-backtrace binds print-escape-control-characters to t, what
> actually gets produced is a mixture of ascii characters and octal
> escaped representations, just like you see when an .elc file is visited
> in Emacs.

Escaping raw bytes doesn't make them more comprehensible.

> > I think you should re-evaluate your preferences, and base them on real
> > advantages and disadvantages, not on imaginary ones.
> 
> What makes you think I'm not doing this already?  I wrote debug-early
> last year because I _needed_ it.  I've a lot of experience using it, and
> the way it prints (or rather used to print) a compiled function is/was
> useful.

For you, maybe.  Not for me.  For me it's a nuisance.  A useless waste
of screen estate.

> You can see how many arguments it takes.

That information exists elsewhere.

> You can see the byte code string, hence enabling you to compare it
> visually with the contents of a .elc file, should such already
> exist.

Let's be serious, okay?

> > If we want our backtraces to be more informative, not less, we should
> > move farther away of "dumb" output functions that just spill the guts
> > and towards more humanly-readable formatted description of the called
> > functions.
> 
> A wholesome long term goal I fully agree with.  For the here and now,
> outputting "#f(compiled-function () <random hex address>" and nothing
> else is not a step in that direction.

But using cl-prin1 _is_ a step in the right direction, because it
allows us to extend the feature.  And that is why I said that if
loading Lisp breaks something in debug-early, we should solve this
_now_, not fall back on prin1, which by its very nature cannot be
extended so easily.

> Besides, the idea of debug-early is to generate a backtrace before all
> the fancy Lisp facilities are available.

Some fancy Lisp facilities are already available anyway.  And I see no
reason for this requirement in this case.

bug#61847: debug-early-backtrace only works some of the time.

[Alan Mackenzie]

Hello, Eli.

On Wed, Mar 01, 2023 at 15:32:35 +0200, Eli Zaretskii wrote:
> > Cc: 61847@debbugs.gnu.org
> > Date: Tue, 28 Feb 2023 19:52:33 +0000
> > From: Alan Mackenzie <acm@muc.de>

> > The purpose of a backtrace is not to enter a beauty contest.  Rather
> > it's to provide the programmer with as much information as
> > reasonably possible to solve a bug.

> Information that is humanly-readable and understandable, yes.  Not
> just any information.  Showing raw bytes of the bytecode is not very
> useful information, IMNSHO.

In my experience, it is useful.  cl-prin1 outputs essentially no
information at all about compiled functions.  That is not useful.

> Or at least we could make it much more useful, if we really want that
> part to be presented to the programmer.

Debugging used to be done with core dumps.  debug-early-backtrace's
output using prin1 is more useful than that.  And yes, we do want to
present it to the programmer.

> > prin1 by contrast prints the actual contents of the function - its byte
> > code string and its constant vector, among other things.  It may not be
> > as "readable", but it is infinitely more useful to the person trying to
> > debug a bug.

> 1 is "infinitely more" than zero, but it is still just 1.  Not 1`00
> and not 1000.

debug-early, using prin1, generates useful output for debugging problems
in early bootstrap.  Using cl-prin1, the output is less useful.

> IOW, just because relatively you get an "infinite" improvement, the
> net improvement is still very small, and there's no reason to stop
> there.

The improvement is significant.

For what it's worth, I think debugging in Emacs, at whatever level, is
currently too hard, too uncertain, and too laborious.  I've been working
to try and improve this for quite a long time.

> So your argument against cl-prin1, if taken to its logical conclusion,
> should be rephrased as "let's improve cl-prin1", not "let's go back to
> the infinitely useless prin1".

My main argument against cl-prin1 is that it's Lisp, and loading Lisp in
early bootstrap is an uncertain, difficult process, as this bug shows.
The guiding design principle in debug-early.el was to use _NO_ Lisp at
all, other than debug-early.el itself.  This was for the sake of the
solidity which temacs has, in comparison with the unprocessed Lisp
source files.

prin1 is _far_ from "infinitely useless".  I've used it to good effect.
cl-prin1 is not useful for printing compiled functions.

How about this suggestion: to fix the bug right now, we put prin1 back
in, as it was in the original debug-early.el.  When cl-prin1 has
improved sufficiently, we then consider putting it back into
debug-early.el?

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#61847: debug-early-backtrace only works some of the time.

[Alan Mackenzie]

Hello, Eli.

On Wed, Mar 01, 2023 at 18:01:24 +0200, Eli Zaretskii wrote:
> > Date: Wed, 1 Mar 2023 15:22:29 +0000
> > Cc: monnier@iro.umontreal.ca, 61847@debbugs.gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > And what use is the meaningless stream of raw bytes that prin1
> > > produces?

> > If that were indeed what was produced, none at all.  Because
> > debug-early-backtrace binds print-escape-control-characters to t, what
> > actually gets produced is a mixture of ascii characters and octal
> > escaped representations, just like you see when an .elc file is visited
> > in Emacs.

> Escaping raw bytes doesn't make them more comprehensible.

It does, if they contain control characters which would otherwise corrupt
the screen.

> > > I think you should re-evaluate your preferences, and base them on real
> > > advantages and disadvantages, not on imaginary ones.

> > What makes you think I'm not doing this already?  I wrote debug-early
> > last year because I _needed_ it.  I've a lot of experience using it, and
> > the way it prints (or rather used to print) a compiled function is/was
> > useful.

> For you, maybe.  Not for me.  For me it's a nuisance.  A useless waste
> of screen estate.

Yes, for me, and lots of other people too.  Byte code is very compact,
and we're talking about a very few lines of screen "wasted".

> > You can see how many arguments it takes.

> That information exists elsewhere.

You're misunderstanding the situation.  You see a lambda form represented
in a backtrace, you don't know which lambda form it is.  There are over
8,000 of them in Emacs.  Knowing the number of arguments helps identify
_which_ lambda form it is.

> > You can see the byte code string, hence enabling you to compare it
> > visually with the contents of a .elc file, should such already
> > exist.

> Let's be serious, okay?

I'm being perfectly serious.  I've done this.  It enabled me to be sure
that I'd identified the lambda form in question, removing the doubt.

> > > If we want our backtraces to be more informative, not less, we
> > > should move farther away of "dumb" output functions that just spill
> > > the guts and towards more humanly-readable formatted description of
> > > the called functions.

> > A wholesome long term goal I fully agree with.  For the here and now,
> > outputting "#f(compiled-function () <random hex address>" and nothing
> > else is not a step in that direction.

> But using cl-prin1 _is_ a step in the right direction, because it
> allows us to extend the feature.

Well then, let's get cl-prin1 up to the state where it is better than
bare prin1.  As I keep saying cl-prin1 is useless for printing compiled
lambda forms at the moment.

> And that is why I said that if loading Lisp breaks something in
> debug-early, we should solve this _now_, not fall back on prin1, which
> by its very nature cannot be extended so easily.

That it cannot be extended easily is not a disadvantage.  It means there
is always a dependable debug-early-backtrace.  The attempt to load
cl-prin1 into debug-early is precisely what caused this bug.

> > Besides, the idea of debug-early is to generate a backtrace before all
> > the fancy Lisp facilities are available.

> Some fancy Lisp facilities are already available anyway.  And I see no
> reason for this requirement in this case.

In a situation where Emacs does not bootstrap, instead crashing during
the loading or compiling of the Lisp files, we need a debugging tool
which isn't dependent upon those Lisp files.  That is what debug-early.el
is intended to be.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#61847: debug-early-backtrace only works some of the time.

[Eli Zaretskii]

> Date: Wed, 1 Mar 2023 16:05:58 +0000
> Cc: monnier@iro.umontreal.ca, 61847@debbugs.gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> My main argument against cl-prin1 is that it's Lisp, and loading Lisp in
> early bootstrap is an uncertain, difficult process, as this bug shows.

My main point, which I evidently fail to drive home, is that we must
do that anyway, if we want these backtraces to show a reasonably
readable information.

> prin1 is _far_ from "infinitely useless".  I've used it to good effect.

Yes, and when I was younger, I used MSDOS to good effect.

> cl-prin1 is not useful for printing compiled functions.

Then let's make it useful!  That's the right direction, not falling
back to prin1.

> How about this suggestion: to fix the bug right now, we put prin1 back
> in, as it was in the original debug-early.el.  When cl-prin1 has
> improved sufficiently, we then consider putting it back into
> debug-early.el?

No.

bug#61847: debug-early-backtrace only works some of the time.

[Eli Zaretskii]

> Date: Wed, 1 Mar 2023 16:46:52 +0000
> Cc: monnier@iro.umontreal.ca, 61847@debbugs.gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > Some fancy Lisp facilities are already available anyway.  And I see no
> > reason for this requirement in this case.
> 
> In a situation where Emacs does not bootstrap, instead crashing during
> the loading or compiling of the Lisp files, we need a debugging tool
> which isn't dependent upon those Lisp files.  That is what debug-early.el
> is intended to be.

You are saying that only Lisp code can crash and make debugging
harder?  Have you never seen crashes inside prin1?

Any code that prints arbitrary objects can crash.  That fact is not a
useful argument for or against a particular alternative for such
printing.

Anyway, I don't see any point in continuing this argument.  It is
clear that we disagree here, and the nature of the disagreement is
also very clear.

bug#61847: debug-early-backtrace only works some of the time.

[Alan Mackenzie]

Hello, Eli.

On Wed, Mar 01, 2023 at 19:04:32 +0200, Eli Zaretskii wrote:
> > Date: Wed, 1 Mar 2023 16:46:52 +0000
> > Cc: monnier@iro.umontreal.ca, 61847@debbugs.gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > Some fancy Lisp facilities are already available anyway.  And I see no
> > > reason for this requirement in this case.

> > In a situation where Emacs does not bootstrap, instead crashing during
> > the loading or compiling of the Lisp files, we need a debugging tool
> > which isn't dependent upon those Lisp files.  That is what debug-early.el
> > is intended to be.

> You are saying that only Lisp code can crash and make debugging
> harder?  Have you never seen crashes inside prin1?

Of course not.  I'm saying that Lisp code, in early bootstrap, DOES
crash, and for that we need a suitable tool, namely debug-early.el.  For
other sorts of crashes we use other tools.  And no, I've never seen a
crash inside prin1.  That function was debugged long before I started
hacking on Emacs.

> Any code that prints arbitrary objects can crash.  That fact is not a
> useful argument for or against a particular alternative for such
> printing.

> Anyway, I don't see any point in continuing this argument.  It is
> clear that we disagree here, and the nature of the disagreement is
> also very clear.

Alright, but we still have a bug to fix.  I think (but I'm not sure)
that you agree that cl-prin1 in its current state isn't currently
adequate for debug-early.el.

You have rejected my proposed fix.  So what alternative do you propose?

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#61847: debug-early-backtrace only works some of the time.

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> the way it prints (or rather used to print) a compiled function is/was
> useful.  You can see how many arguments it takes.

Hmm...

`prin1` will print for example #[256 ...] where 256 is where it tells
"how many arguments it takes".
In contrast, `cl-prin1` will print #f(compiled-function (&optional arg) ...)

So in both cases "you can see how many arguments it takes" (and I'll
let you guess which one I find more readable).

FWIW, I just tried the `condition-case` wrapper and it fixes the problem
for the test case you showed.  So I just pushed my patch to
`emacs-29` (which additionally checks for the presence of
`cl-defmethod`, which also fixes the problem: one is never too sure).

I'll let you (plural) decide whether to keep using `cl-prin1` or not,
but now this is not directly related to fixing this bug any more.


        Stefan

bug#61847: debug-early-backtrace only works some of the time.

[Eli Zaretskii]

> Date: Wed, 1 Mar 2023 17:31:23 +0000
> Cc: monnier@iro.umontreal.ca, 61847@debbugs.gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> You have rejected my proposed fix.  So what alternative do you propose?

To start from what Stefan suggested, and improve/fix that if needed.

bug#61847: debug-early-backtrace only works some of the time.

[Alan Mackenzie]

Hello, Stefan.

On Wed, Mar 01, 2023 at 12:34:04 -0500, Stefan Monnier wrote:
> > the way it prints (or rather used to print) a compiled function is/was
> > useful.  You can see how many arguments it takes.

> Hmm...

> `prin1` will print for example #[256 ...] where 256 is where it tells
> "how many arguments it takes".
> In contrast, `cl-prin1` will print #f(compiled-function (&optional arg) ...)

> So in both cases "you can see how many arguments it takes" (and I'll
> let you guess which one I find more readable).

I find them both equally readable, having practised reading dumped
functions.  But I accept most people won't.

> FWIW, I just tried the `condition-case` wrapper and it fixes the problem
> for the test case you showed.  So I just pushed my patch to
> `emacs-29` (which additionally checks for the presence of
> `cl-defmethod`, which also fixes the problem: one is never too sure).

One is never too sure, which was one of the points of the original
design.

> I'll let you (plural) decide whether to keep using `cl-prin1` or not,
> but now this is not directly related to fixing this bug any more.

You could have reminded me of the variable cl-print-compiled, which at
least allows you to dump the constant vector.

I propose extending this variable thusly:


diff --git a/lisp/emacs-lisp/cl-print.el b/lisp/emacs-lisp/cl-print.el
index 61586526ca1..74dad303a27 100644
--- a/lisp/emacs-lisp/cl-print.el
+++ b/lisp/emacs-lisp/cl-print.el
@@ -165,6 +165,7 @@ 'help-byte-code
 (defvar cl-print-compiled nil
   "Control how to print byte-compiled functions.
 Acceptable values include:
+- `full' to print out the full contents of the function using `prin1'.
 - `static' to print the vector of constants.
 - `disassemble' to print the disassembly of the code.
 - nil to skip printing any details about the code.")

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).