bug#11218: with-demoted-errors use of condition-case-unless-debug; ert

[Christian Ohler <ohler@gnu.org>]

Stefan Monnier, 2012-04-19:

>>>>>> If you replace with-demoted-errors with ignore-errors, the test passes.
>>>>> Looks like a bug in ERT.
>>>> I don't know if it's a "bug" per se...
>>>> ert--run-test-internal binds debug-on-error to t, and redefines the
>>>> debugger to ert--run-test-debugger. As the doc of that function says:
>>> I wonder why ERT doesn't just use condition-case to catch and record
>>> the errors.
>> Two reasons that I remember off the top of my head: Recording backtraces,
>> and recording additional information provided with `ert-info'.
>
> So the next question is: why does ERT record a backtrace and that extra
> information provided by ert-info?

To be able to show them to the user, together with the test failure.  If 
you have one or more failing tests, it's much more useful to see the 
error messages and backtraces rather than just the error messages.

Unless there's another way to accomplish this, I think it makes sense 
for ERT to bind `debugger'.  Since `debugger' is only called if 
`debug-on-error' is set, ERT has to bind that, too.

I'm not sure where that leaves us.  You could say the problem is that 
`debug-on-error' is overloaded with two different meanings (one in 
eval.c, one in `condition-case-no-debug'), or you could say that ERT 
wants a different `debugger' hook that does not depend on `debug-on-errors'.

But perhaps `with-demoted-errors' and its interaction with 
`debug-on-error' is inherently problematic -- how would we like it to 
interact with tests and debugging?  Assuming ERT did not rebind 
`debug-on-error', the test

   (ert-deftest foo ()
     (with-demoted-errors (error "a"))
     (error "b"))

would normally fail with "b", but if we enable debugging to track down 
why, it will fail with "a" instead (with no way to continue execution to 
get to the error we are interested in).  Confusing.  As far as ERT tests 
are concerned, one could argue that the current behavior (always fail 
with "a") is preferable because it's more consistent; it just means that 
errors in tests can't be demoted.

Fundamentally, having conditionals in the code that explicitly check 
`debug-on-error' seems like asking for trouble.  If the program checks 
"am I being debugged" and behaves differently in that case, we should 
expect testing and debugging tools to break...

And ERT is not the only tool that triggers the problem: While 
investigating this bug, I set `debug-on-error' to nil globally, typed 
M-: (with-demoted-errors (error "foo")) RET, and was confused for a 
moment that I found myself in the debugger with error "foo" -- 
`with-demoted-errors' had no effect.  This is because M-: rebinds 
`debug-on-error' (at least by default).  Just like ERT.  Edebug does the 
same, too.

Christian.