Re: master bb9133f09da: * lisp/emacs-lisp/ert.el (ert-run-tests-batch-and-exit): Inhibit interaction

Re: master bb9133f09da: * lisp/emacs-lisp/ert.el (ert-run-tests-batch-and-exit): Inhibit interaction

[Michael Albinus]

Stefan Monnier via Mailing list for Emacs changes <emacs-diffs@gnu.org>
writes:

Hi Stefan,

>     * lisp/emacs-lisp/ert.el (ert-run-tests-batch-and-exit): Inhibit interaction

This let fail several dozen tests on emba. Please revert.

Best regards, Michael.

Re: master bb9133f09da: * lisp/emacs-lisp/ert.el (ert-run-tests-batch-and-exit): Inhibit interaction

[Mattias Engdegård]

14 aug. 2023 kl. 09.13 skrev Michael Albinus <michael.albinus@gmx.de>:

>>    * lisp/emacs-lisp/ert.el (ert-run-tests-batch-and-exit): Inhibit interaction
> 
> This let fail several dozen tests on emba. Please revert.

Reverted. Thank you for noticing.

This is something anyone can do, at least in clear-cut situations like this one. If in doubt, ask. (Which you did, have another gold star!)

Re: master bb9133f09da: * lisp/emacs-lisp/ert.el (ert-run-tests-batch-and-exit): Inhibit interaction

[Stefan Monnier]

>>     * lisp/emacs-lisp/ert.el (ert-run-tests-batch-and-exit): Inhibit interaction
> This let fail several dozen tests on emba. Please revert.

Hmm... why?

FWIW, I added it because I'm seeing tests (e.g. `file-tests.el`) hang in
`read-event` (apparently because a (cl-letf ((symbol-function ..)) ..)
rebinding is not taking effect :-( ) and I figured that batch tests
should indeed never need interaction.


        Stefan

Re: master bb9133f09da: * lisp/emacs-lisp/ert.el (ert-run-tests-batch-and-exit): Inhibit interaction

[Mattias Engdegård]

14 aug. 2023 kl. 16.00 skrev Stefan Monnier <monnier@iro.umontreal.ca>:

> FWIW, I added it because I'm seeing tests (e.g. `file-tests.el`) hang in
> `read-event` (apparently because a (cl-letf ((symbol-function ..)) ..)
> rebinding is not taking effect :-( ) and I figured that batch tests
> should indeed never need interaction.

I agree, but some tests persist. For example, abbrev-tests uses ert-simulate-keys to 'interact' with Emacs.
Other test do morally equivalent things, and it is indubitably useful to be able to test interactive features non-interactively in some way.

Re: master bb9133f09da: * lisp/emacs-lisp/ert.el (ert-run-tests-batch-and-exit): Inhibit interaction

[Stefan Monnier]

>> FWIW, I added it because I'm seeing tests (e.g. `file-tests.el`) hang in
>> `read-event` (apparently because a (cl-letf ((symbol-function ..)) ..)
>> rebinding is not taking effect :-( ) and I figured that batch tests
>> should indeed never need interaction.
>
> I agree, but some tests persist. For example, abbrev-tests uses
> ert-simulate-keys to 'interact' with Emacs.
> Other test do morally equivalent things, and it is indubitably useful to be
> able to test interactive features non-interactively in some way.

Oh, I think I see: `inhibit-interaction` is probably tested too early,
e.g. upon entry to `read-event` (which can return without user
interaction if there's enough `unread-command-events` and friends)
instead of upon entry to the actual code that waits for user input, so
it doesn't quite do what I thought it would.


        Stefan

Re: master bb9133f09da: * lisp/emacs-lisp/ert.el (ert-run-tests-batch-and-exit): Inhibit interaction

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

Hi Stefan,

> Oh, I think I see: `inhibit-interaction` is probably tested too early,
> e.g. upon entry to `read-event` (which can return without user
> interaction if there's enough `unread-command-events` and friends)
> instead of upon entry to the actual code that waits for user input, so
> it doesn't quite do what I thought it would.

If you want to check yourself, here are the tests which failed due to
the (inhibited-interaction) check on emba:

--8<---------------cut here---------------start------------->8---
   FAILED  50/91  tramp-test26-interactive-file-name-completion (0.004941 sec) at lisp/net/tramp-tests.el:4841
   FAILED   1/22  abbrev--possibly-save-test (0.004799 sec) at lisp/abbrev-tests.el:308
   FAILED  1/7  auto-revert-test00-auto-revert-mode (0.215386 sec) at lisp/autorevert-tests.el:146
   FAILED  2/7  auto-revert-test01-auto-revert-several-files (0.385464 sec) at lisp/autorevert-tests.el:191
   FAILED  3/7  auto-revert-test03-auto-revert-tail-mode (0.033018 sec) at lisp/autorevert-tests.el:335
   FAILED  4/7  auto-revert-test04-auto-revert-mode-dired (0.098083 sec) at lisp/autorevert-tests.el:370
   FAILED  6/7  auto-revert-test06-write-file (0.035447 sec) at lisp/autorevert-tests.el:538
   FAILED  7/7  auto-revert-test07-auto-revert-several-buffers (0.042405 sec) at lisp/autorevert-tests.el:572
   FAILED  2/6  find-func-tests--library-completion (0.001559 sec) at lisp/emacs-lisp/find-func-tests.el:32
   FAILED  4/5  viper-test-undo-3 (0.057615 sec) at lisp/emulation/viper-tests.el:118
   FAILED  1/5  erc-add-entry-to-list (0.000351 sec) at lisp/erc/erc-match-tests.el:27
   FAILED  11/15  erc-nicks-customize-face (0.127834 sec) at lisp/erc/erc-nicks-tests.el:441
   FAILED  3/9  erc-sasl--read-password--basic (0.029777 sec) at lisp/erc/erc-sasl-tests.el:35
   FAILED  17/17  erc-nickserv-get-password (0.002919 sec) at lisp/erc/erc-services-tests.el:534
   FAILED   4/54  erc--check-prompt-input-for-excess-lines (0.000534 sec) at lisp/erc/erc-tests.el:1187
   FAILED  13/54  erc--interactive (0.002051 sec) at lisp/erc/erc-tests.el:1829
   FAILED  26/54  erc--switch-to-buffer (0.001627 sec) at lisp/erc/erc-tests.el:465
   FAILED  46/54  erc-select-read-args (0.000371 sec) at lisp/erc/erc-tests.el:1606
   FAILED  50/54  erc-server-select (0.008819 sec) at lisp/erc/erc-tests.el:1865
   FAILED  52/54  erc-tls (0.058334 sec) at lisp/erc/erc-tests.el:1713
   FAILED   5/18  file-notify-test02-rm-watch (0.045676 sec) at lisp/filenotify-tests.el:396
   FAILED   6/18  file-notify-test02-rm-watch-remote (3.690951 sec) at lisp/filenotify-tests.el:1
   FAILED   7/18  file-notify-test04-autorevert (2.026364 sec) at lisp/filenotify-tests.el:937
   FAILED   8/18  file-notify-test05-file-validity (0.023406 sec) at lisp/filenotify-tests.el:1039
   FAILED   9/18  file-notify-test05-file-validity-remote (3.579880 sec) at lisp/filenotify-tests.el:1
   FAILED  12/18  file-notify-test07-many-events (0.023911 sec) at lisp/filenotify-tests.el:1220
   FAILED  13/18  file-notify-test08-backup (0.009403 sec) at lisp/filenotify-tests.el:1298
   FAILED  14/18  file-notify-test08-backup-remote (1.419481 sec) at lisp/filenotify-tests.el:1
   FAILED  17/18  file-notify-test11-symlinks (0.011308 sec) at lisp/filenotify-tests.el:1581
   FAILED  18/18  file-notify-test11-symlinks-remote (1.899609 sec) at lisp/filenotify-tests.el:1
   FAILED   2/13  mule-cmds--test-universal-coding-system-argument (0.000393 sec) at lisp/international/mule-tests.el:45
   FAILED  25/44  kmacro-tests-step-edit-edits-macro-with-extended-command (0.001692 sec) at lisp/kmacro-tests.el:800
   FAILED   3/25  completion-auto-help-test (0.000958 sec) at lisp/minibuffer-tests.el:346
   FAILED   4/25  completion-auto-select-test (0.000783 sec) at lisp/minibuffer-tests.el:374
   FAILED   5/25  completion-auto-wrap-test (0.000320 sec) at lisp/minibuffer-tests.el:396
   FAILED  24/25  completions-affixation-navigation-test (0.000354 sec) at lisp/minibuffer-tests.el:453
   FAILED  25/25  completions-header-format-test (0.000346 sec) at lisp/minibuffer-tests.el:425
   FAILED   4/48  eglot-test-basic-completions (0.363706 sec) at lisp/progmodes/eglot-tests.el:540
   FAILED   5/48  eglot-test-basic-diagnostics (0.280194 sec) at lisp/progmodes/eglot-tests.el:421
   FAILED  12/48  eglot-test-eldoc-after-completions (0.440281 sec) at lisp/progmodes/eglot-tests.el:639
   FAILED  14/48  eglot-test-formatting (0.424955 sec) at lisp/progmodes/eglot-tests.el:672
   FAILED  19/48  eglot-test-multiline-eldoc (0.334466 sec) at lisp/progmodes/eglot-tests.el:655
   FAILED  20/48  eglot-test-non-unique-completions (0.290398 sec) at lisp/progmodes/eglot-tests.el:556
   FAILED  19/29  query-replace-search-function-tests (0.009815 sec) at lisp/replace-tests.el:459
   FAILED  20/29  query-replace-tests (0.000823 sec) at lisp/replace-tests.el:411
   FAILED   2/13  ses-jump-B2-lowcase-keys (0.010933 sec) at lisp/ses-tests.el:213
   FAILED  34/48  simple-test-undo-extra-boundary-in-tex (0.078750 sec) at lisp/simple-tests.el:694
   FAILED   2/10  whitespace-tests--empty-bob (0.001325 sec) at lisp/whitespace-tests.el:130
   FAILED   5/10  whitespace-tests--empty-eob (0.001189 sec) at lisp/whitespace-tests.el:232
   FAILED  1/3  xt-mouse-tracking-basic (0.412822 sec) at lisp/xt-mouse-tests.el:60
   FAILED  2/3  xt-mouse-tracking-sgr (0.001347 sec) at lisp/xt-mouse-tests.el:101
   FAILED  3/3  xt-mouse-tracking-utf-8 (0.001496 sec) at lisp/xt-mouse-tests.el:79
   FAILED  1/2  inotify-file-watch-simple (0.008711 sec) at src/inotify-tests.el:52
   FAILED  2/2  keyboard-unread-command-events (0.000233 sec) at src/keyboard-tests.el:24
   FAILED  27/32  threads-signal-main-thread (0.002614 sec) at src/thread-tests.el:325
   FAILED   6/10  xdisp-tests--minibuffer-resizing (0.007032 sec) at src/xdisp-tests.el:35
   FAILED   7/10  xdisp-tests--minibuffer-scroll (0.000490 sec) at src/xdisp-tests.el:55
--8<---------------cut here---------------end--------------->8---

>         Stefan

Best regards, Michael.

Re: master bb9133f09da: * lisp/emacs-lisp/ert.el (ert-run-tests-batch-and-exit): Inhibit interaction

[Stefan Monnier]

> Oh, I think I see: `inhibit-interaction` is probably tested too early,
> e.g. upon entry to `read-event` (which can return without user
> interaction if there's enough `unread-command-events` and friends)
> instead of upon entry to the actual code that waits for user input, so
> it doesn't quite do what I thought it would.

And even if we move the check to the actual place in `read_char` where
we wait for user input, it's still too early because it burps in those
places in the tests where we `read-event` only for the timeout (i.e. we
do wait, but no user-interaction is needed).


        Stefan

Re: master bb9133f09da: * lisp/emacs-lisp/ert.el (ert-run-tests-batch-and-exit): Inhibit interaction

[Jens Schmidt]

On 2023-08-14  16:00, Stefan Monnier wrote:

> FWIW, I added it because I'm seeing tests (e.g. `file-tests.el`) hang in
> `read-event` (apparently because a (cl-letf ((symbol-function ..)) ..)
> rebinding is not taking effect :-( )

OT: The hanging tests might be related to bug #65176, also discussed in
the thread starting here:

  https://lists.gnu.org/archive/html/emacs-devel/2023-08/msg00311.html

Re: master bb9133f09da: * lisp/emacs-lisp/ert.el (ert-run-tests-batch-and-exit): Inhibit interaction

[Stefan Monnier]

>> FWIW, I added it because I'm seeing tests (e.g. `file-tests.el`) hang in
>> `read-event` (apparently because a (cl-letf ((symbol-function ..)) ..)
>> rebinding is not taking effect :-( )
> OT: The hanging tests might be related to bug #65176, also discussed in
> the thread starting here:

Yup.  My commit was intended to replace the hangs with "cleaner" errors
when such problems occur in the future.


        Stefan