bug#75357: [scratch/elisp-benchmarks] elb-scroll window size assumption breaks on repeated runs

bug#75357: [scratch/elisp-benchmarks] elb-scroll window size assumption breaks on repeated runs

[Pip Cet via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On scratch/elisp-benchmarks, the following emacs -Q invocation:

./src/emacs --batch -Q -l ./elisp-benchmarks/elisp-benchmarks.el --eval '(progn (elisp-benchmarks-run "scroll" t 1) (elisp-benchmarks-run "scroll" t 1))'

produces a fatal Lisp error, attempting to invoke the debugger in a
non-interactive session:

Loading /home/pip/emacs-elisp-benchmarks/elisp-benchmarks/benchmarks/elb-smie...
Window size: 19 x 80
Debugger entered--Lisp error: (error "Window size not as stipulated by the benchmark")
  signal(error ("Window size not as stipulated by the benchmark"))
  error("Window size not as stipulated by the benchmark")
  elb-scroll-entry()

There are several bugs here:

1. The test itself incorrectly assumes that no secondary window is
displayed which would affect the ability of the test to resize its
window to the specified dimensions.

2. elisp-benchmarks does not handle Lisp errors thrown by a test
gracefully if debug-on-error is initially t, since it uses
with-demoted-errors without let-binding debug-on-error.

3. elisp-benchmarks destructively uses setq to modify the session's
debug-on-error value, leaving it at a different value on exit.

(1) can be fixed by a call to (delete-other-windows) in noninteractive
mode, but cannot be fixed that way in interactive mode.

(2) can be fixed by let-binding debug-on-error in non-interactive mode,
or using condition-case properly.

(3) can and should be fixed in the obvious fashion.

(2) and (3) can also be fixed by moving to ERT, which has an established
API for handling both the interactive and non-interactive scenario.

In all cases, we need a way to indicate for a benchmark whether it
modifies Emacs state destructively.  We need to handle debug-on-quit
differently, as ERT does.

bug#75357: [scratch/elisp-benchmarks] elb-scroll window size assumption breaks on repeated runs

[Andrea Corallo]

Pip Cet via "Bug reports for GNU Emacs, the Swiss army knife of text
editors" <bug-gnu-emacs@gnu.org> writes:

> On scratch/elisp-benchmarks, the following emacs -Q invocation:
>
> ./src/emacs --batch -Q -l ./elisp-benchmarks/elisp-benchmarks.el --eval '(progn (elisp-benchmarks-run "scroll" t 1) (elisp-benchmarks-run "scroll" t 1))'
>
> produces a fatal Lisp error, attempting to invoke the debugger in a
> non-interactive session:
>
> Loading /home/pip/emacs-elisp-benchmarks/elisp-benchmarks/benchmarks/elb-smie...
> Window size: 19 x 80
> Debugger entered--Lisp error: (error "Window size not as stipulated by the benchmark")
>   signal(error ("Window size not as stipulated by the benchmark"))
>   error("Window size not as stipulated by the benchmark")
>   elb-scroll-entry()

I'm not sure this is a bug as we want always to run the benchmark from a
known state, anyway adding Stefan as he's the author of the benchmark.

bug#75357: [scratch/elisp-benchmarks] elb-scroll window size assumption breaks on repeated runs

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> On scratch/elisp-benchmarks, the following emacs -Q invocation:
>>
>> ./src/emacs --batch -Q -l ./elisp-benchmarks/elisp-benchmarks.el --eval '(progn (elisp-benchmarks-run "scroll" t 1) (elisp-benchmarks-run "scroll" t 1))'
>>
>> produces a fatal Lisp error, attempting to invoke the debugger in a
>> non-interactive session:
>>
>> Loading /home/pip/emacs-elisp-benchmarks/elisp-benchmarks/benchmarks/elb-smie...
>> Window size: 19 x 80
>> Debugger entered--Lisp error: (error "Window size not as stipulated by the benchmark")
>>   signal(error ("Window size not as stipulated by the benchmark"))
>>   error("Window size not as stipulated by the benchmark")
>>   elb-scroll-entry()
>
> I'm not sure this is a bug as we want always to run the benchmark from a
> known state, anyway adding Stefan as he's the author of the benchmark.

Sounds like something has changed in Emacs's C code, or maybe it's
something in some system-depend code?
`elb-scroll` cares about the window size because it's a benchmark that
is influenced by (and thus partly measures) the redisplay:

    (unless (and noninteractive (not (boundp 'redisplay-skip-initial-frame)))
      (defun elb-scroll-entry ()
        ;; FIXME: This relies on `elb-smie.el' being compiled already which is
        ;; not necessarily the case if we're only running some of the benchmarks.
        (load (expand-file-name "elb-smie" elb-bench-directory) nil 'nomessage)
        (setq redisplay-skip-initial-frame nil)
        (with-temp-buffer
          (rename-buffer (generate-new-buffer-name "elb-scroll"))
          (switch-to-buffer (current-buffer))
          (insert-file-contents (expand-file-name
                                 "../resources/xmenu.c" elb-bench-directory))
          (redisplay 'force) ;; Refresh the window dimensions.
          (enlarge-window (- 23 (window-height)))
          (enlarge-window (- 80 (window-width)) 'horiz)
          (redisplay 'force) ;; Refresh the window dimensions.
          (unless (and (equal 23 (window-height))
                       (equal 80 (window-width)))
            (error "Window size %S x %S not as stipulated by the benchmark"
                    (window-height) (window-width)))
          (dotimes (_ 10)
            (elb-smie-mode)
            (goto-char (point-min))
            (condition-case nil
                (while t (scroll-up nil) (redisplay 'force))
              (end-of-buffer nil))))))

Try and figure out why window size is 19x80 after

          (enlarge-window (- 23 (window-height)))
          (enlarge-window (- 80 (window-width)) 'horiz)

- Stefan