bug#5803: 23.1.94; Using tq-enqueue in timer messes with keystroke echoing

bug#5803: 23.1.94; Using tq-enqueue in timer messes with keystroke echoing

[David Engster]

This bug report will be sent to the Free Software Foundation,
not to your local site managers!
Please write in English if possible, because the Emacs maintainers
usually do not have translators to read other languages for them.

Your bug report will be posted to the bug-gnu-emacs@gnu.org mailing list,
and to the gnu.emacs.bug news group.

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug.  If you can, give
a recipe starting from `emacs -Q':

When using `tq-enqueue' in a timer, the keystroke echoing in the
echo area becomes weird or is completely invisible.  You should be able
to reproduce this as follows:

* emacs -Q

* Evaluate the following:

(setq proc (start-process "shell" (get-buffer-create "*process*") "/bin/sh"))
(setq mytq (tq-create proc))
(defun send-test ()
  (tq-enqueue mytq "echo TEST\n" "TEST[^0]*" nil nil))
(setq timer (run-with-timer 1 1 'send-test))

* Now hit 'C-x' and wait a second.

* The string "C-x-" should appear in the echo area, but nothing can be
  seen.

* Hit "C-g". (The message "C-x C-g is undefined" appears, which is
  expected.)

* Now evaluate

(cancel-timer timer)
(setq timer (run-with-timer 1 2 'send-test))

* Wait a few seconds, now again hit "C-x" and watch the echo area.

* In seemingly random succession, the strings "C-x", " C-x" and the
  empty string appear, with a few seconds time in between.

* Evaluating (cancel-timer timer) will restore normal behavior.


If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
For information about debugging Emacs, please read the file
/opt/emacs/share/emacs/23.1.94/etc/DEBUG.


In GNU Emacs 23.1.94.1 (i686-pc-linux-gnu, GTK+ Version 2.18.3)
 of 2010-03-30 on void
Windowing system distributor `The X.Org Foundation', version 11.0.10604000
configured using `configure  '--prefix=/opt/emacs''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Emacs-Lisp

Minor modes in effect:
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
M-x v e r s i o n <return> C-x C-f [ r p c y <backspace> 
<backspace> <backspace> <backspace> <backspace> p r 
o c <tab> <return> <down> <down> <down> <down> <down> 
<down> C-k C-k <down> <down> <down> <up> <up> <down> 
<down> <down> <down> <right> <right> <left> <left> 
<up> <up> <up> <up> <up> <left> C-x C-e <down> C-x 
C-e <down> <down> C-e C-x C-e <down> C-x C-e <down> 
C-e C-x C-g <down> <up> C-x C-e <down> <down> C-x C-g 
<up> <up> <up> <down> <down> <down> <left> C-x C-e 
<up> <up> <down> <right> <right> <right> <right> <right> 
<right> <right> <right> <right> <backspace> 2 C-e C-x 
C-e <down> <down> C-x <left> C-x C-x b C-x b <left> 
<return> <left> C-x C-e <up> <up> <down> <right> <right> 
<right> <right> <right> <right> <right> <right> <right> 
<backspace> 3 C-e C-x C-e <down> C-x C-g M-x r e p 
o <tab> r <tab> <return>

Recent messages:
(nil #<process shell> . #<buffer  tq-temp-shell>)
send-test
nil
[nil 19377 10445 698196 1 send-test nil nil]
nil
[nil 19377 10458 258144 2 send-test nil nil]
exchange-point-and-mark: No mark set in this buffer
nil
[nil 19377 10482 122182 3 send-test nil nil]
Making completion list...

Load-path shadows:
None found.

Features:
(shadow sort mail-extr message sendmail regexp-opt ecomplete rfc822 mml
mml-sec password-cache mm-decode mm-bodies mm-encode mailcap mail-parse
rfc2231 rfc2047 rfc2045 qp ietf-drums mailabbrev nnheader gnus-util
netrc time-date mm-util mail-prsvr gmm-utils wid-edit mailheader canlock
sha1 hex-util hashcash mail-utils emacsbug help-mode easymenu view tq
tooltip ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd
font-setting tool-bar dnd fontset image fringe lisp-mode register page
menu-bar rfn-eshadow timer select scroll-bar mldrag mouse jit-lock
font-lock syntax facemenu font-core frame cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese hebrew
greek romanian slovak czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple abbrev loaddefs button
minibuffer faces cus-face files text-properties overlay md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process dbusbind system-font-setting
font-render-setting gtk x-toolkit x multi-tty emacs)

bug#5803: 23.1.94; Using tq-enqueue in timer messes with keystroke echoing

[Lars Ingebrigtsen]

David Engster <deng@randomsample.de> writes:

> When using `tq-enqueue' in a timer, the keystroke echoing in the
> echo area becomes weird or is completely invisible.  You should be able
> to reproduce this as follows:
>
> * emacs -Q
>
> * Evaluate the following:
>
> (setq proc (start-process "shell" (get-buffer-create "*process*") "/bin/sh"))
> (setq mytq (tq-create proc))
> (defun send-test ()
>   (tq-enqueue mytq "echo TEST\n" "TEST[^0]*" nil nil))
> (setq timer (run-with-timer 1 1 'send-test))
>
> * Now hit 'C-x' and wait a second.
>
> * The string "C-x-" should appear in the echo area, but nothing can be
>   seen.

(This bug report unfortunately got no response at the time.)

I can reproduce this in Emacs 28 with this small change:

(setq proc (start-process "shell" (get-buffer-create "*process*") "/bin/sh"))
(setq mytq (tq-create proc))
(defun send-test ()
  (tq-enqueue mytq "echo TEST\n" "TEST[^0]*" nil 'ignore))
(setq timer (run-with-timer 1 1 'send-test))

I'm guessing this somehow interferes with the `echo-timeout' thing?
(Changing it to something shorter than 1 second makes the C-x-
appear...  sometimes.)

Anybody got any insight into what might be happening here?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#5803: 23.1.94; Using tq-enqueue in timer messes with keystroke echoing

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 08 Dec 2020 19:02:51 +0100
> Cc: 5803@debbugs.gnu.org
> 
> (setq proc (start-process "shell" (get-buffer-create "*process*") "/bin/sh"))
> (setq mytq (tq-create proc))
> (defun send-test ()
>   (tq-enqueue mytq "echo TEST\n" "TEST[^0]*" nil 'ignore))
> (setq timer (run-with-timer 1 1 'send-test))
> 
> I'm guessing this somehow interferes with the `echo-timeout' thing?
> (Changing it to something shorter than 1 second makes the C-x-
> appear...  sometimes.)
> 
> Anybody got any insight into what might be happening here?

The sit_for call ends prematurely because output from a subprocess
arrives?

bug#5803: 23.1.94; Using tq-enqueue in timer messes with keystroke echoing

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Anybody got any insight into what might be happening here?
>
> The sit_for call ends prematurely because output from a subprocess
> arrives?

I guess?  But is that supposed to happen?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#5803: 23.1.94; Using tq-enqueue in timer messes with keystroke echoing

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: deng@randomsample.de,  5803@debbugs.gnu.org
> Date: Wed, 09 Dec 2020 13:38:53 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Anybody got any insight into what might be happening here?
> >
> > The sit_for call ends prematurely because output from a subprocess
> > arrives?
> 
> I guess?  But is that supposed to happen?

I vaguely remember some issue with this, I will have to take a closer
look.

bug#5803: 23.1.94; Using tq-enqueue in timer messes with keystroke echoing

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 08 Dec 2020 19:02:51 +0100
> Cc: 5803@debbugs.gnu.org
> 
> I can reproduce this in Emacs 28 with this small change:
> 
> (setq proc (start-process "shell" (get-buffer-create "*process*") "/bin/sh"))
> (setq mytq (tq-create proc))
> (defun send-test ()
>   (tq-enqueue mytq "echo TEST\n" "TEST[^0]*" nil 'ignore))
> (setq timer (run-with-timer 1 1 'send-test))

When I run this, I get an error message about spurious
communications.  Does it work cleanly for you?

Do we really need this tq package to reproduce the issue?  It adds a
layer of complexity to a scenario that is complex to begin with.  I
hope there can be a recipe that avoids using tq.

bug#5803: 23.1.94; Using tq-enqueue in timer messes with keystroke echoing

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> When I run this, I get an error message about spurious
> communications.  Does it work cleanly for you?

I get no errors when testing that code...

> Do we really need this tq package to reproduce the issue?  It adds a
> layer of complexity to a scenario that is complex to begin with.  I
> hope there can be a recipe that avoids using tq.

Yeah, the tq is unnecessary.  The following reproduces for me:

(setq proc (start-process "shell" (get-buffer-create "*process*") "/bin/sh"))
(defun send-test ()
  (process-send-string proc "ls /tmp\n"))
(setq timer (run-with-timer 1 1 'send-test))

I then hit

C-x

and I don't get the "C-x-" prompt.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#5803: 23.1.94; Using tq-enqueue in timer messes with keystroke echoing

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 5803@debbugs.gnu.org,  deng@randomsample.de
> Date: Wed, 09 Dec 2020 19:42:59 +0100
> 
> (setq proc (start-process "shell" (get-buffer-create "*process*") "/bin/sh"))
> (defun send-test ()
>   (process-send-string proc "ls /tmp\n"))
> (setq timer (run-with-timer 1 1 'send-test))
> 
> I then hit
> 
> C-x
> 
> and I don't get the "C-x-" prompt.

Thanks.

I found the discussion about a related issue we had in the past, it
starts here:

  https://lists.gnu.org/archive/html/emacs-devel/2018-09/msg00863.html

and the analysis is here:

  https://lists.gnu.org/archive/html/emacs-devel/2018-09/msg00899.html

That patch was installed, but it only fixes the while-no-input
problem.  It doesn't fix this problem, which has the same root cause:
the buffer-switch event we inject when output from a subprocess
arrives is considered valid input, and so when sit_for makes this
test:

  return detect_input_pending () ? Qnil : Qt;  <<<<<<<<<<<<<<<<<<<

it returns nil, and the keystroke echo is not displayed, because the
logic in keyboard.c assumes this means the user typed something.

So I think one way of fixing this is to make the test at the end of
sit_for smarter: if the only event in the input queue is the
buffer-change event, ignore it and return non-nil.  (Alternatively, we
could do this in read_char, where sit_for is called.)

As yet another alternative, we could ask Stefan whether the patch he
suggested back then is still being used, and if so, whether there were
any adverse effects.  He promised to report back in

  https://lists.gnu.org/archive/html/emacs-devel/2018-09/msg00914.html

but AFAIR never did...

bug#5803: 23.1.94; Using tq-enqueue in timer messes with keystroke echoing

[Stefan Monnier]

> As yet another alternative, we could ask Stefan whether the patch he
> suggested back then is still being used, and if so, whether there were
> any adverse effects.  He promised to report back in
>
>   https://lists.gnu.org/archive/html/emacs-devel/2018-09/msg00914.html
>
> but AFAIR never did...

Yes, I'm still using this patch in my daily Emacs.
And no I haven't noticed any effect.  Of course, that's far from
a guarantee that there are no such adverse effects (while I'm
undoubtedly the most important person in my world, I don't think that's
enough to make a statistically significant sample; and on top of that,
I might very well not have noticed some of the effects).


        Stefan

bug#5803: 23.1.94; Using tq-enqueue in timer messes with keystroke echoing

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  5803@debbugs.gnu.org,
>   deng@randomsample.de
> Date: Thu, 10 Dec 2020 12:29:00 -0500
> 
> >   https://lists.gnu.org/archive/html/emacs-devel/2018-09/msg00914.html
> >
> > but AFAIR never did...
> 
> Yes, I'm still using this patch in my daily Emacs.
> And no I haven't noticed any effect.  Of course, that's far from
> a guarantee that there are no such adverse effects (while I'm
> undoubtedly the most important person in my world, I don't think that's
> enough to make a statistically significant sample; and on top of that,
> I might very well not have noticed some of the effects).

Then perhaps we should install that change and see what breaks?

bug#5803: 23.1.94; Using tq-enqueue in timer messes with keystroke echoing

[Stefan Monnier]

>> >   https://lists.gnu.org/archive/html/emacs-devel/2018-09/msg00914.html
>> > but AFAIR never did...
>> Yes, I'm still using this patch in my daily Emacs.
>> And no I haven't noticed any effect.  Of course, that's far from
>> a guarantee that there are no such adverse effects (while I'm
>> undoubtedly the most important person in my world, I don't think that's
>> enough to make a statistically significant sample; and on top of that,
>> I might very well not have noticed some of the effects).
> Then perhaps we should install that change and see what breaks?

Since this patch mostly just removes code, I'll be delighted to install it.
Should I take this question as an actual "go ahead"?


        Stefan

bug#5803: 23.1.94; Using tq-enqueue in timer messes with keystroke echoing

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: larsi@gnus.org,  5803@debbugs.gnu.org,  deng@randomsample.de
> Date: Thu, 10 Dec 2020 13:40:17 -0500
> 
> > Then perhaps we should install that change and see what breaks?
> 
> Since this patch mostly just removes code, I'll be delighted to install it.
> Should I take this question as an actual "go ahead"?

Let's wait for Lars to chime in, after he reads the past discussions
on this.

bug#5803: 23.1.94; Using tq-enqueue in timer messes with keystroke echoing

[Eli Zaretskii]

> Date: Thu, 10 Dec 2020 20:49:52 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: larsi@gnus.org, 5803@debbugs.gnu.org, deng@randomsample.de
> 
> > From: Stefan Monnier <monnier@iro.umontreal.ca>
> > Cc: larsi@gnus.org,  5803@debbugs.gnu.org,  deng@randomsample.de
> > Date: Thu, 10 Dec 2020 13:40:17 -0500
> > 
> > > Then perhaps we should install that change and see what breaks?
> > 
> > Since this patch mostly just removes code, I'll be delighted to install it.
> > Should I take this question as an actual "go ahead"?
> 
> Let's wait for Lars to chime in, after he reads the past discussions
> on this.

Btw, does this mean the recipe Lars posted here in

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=5803#23

works as expected in your patched Emacs, i.e. you do see the "C-x"
echo after about a second?  Because if not, there's something else at
work here, and installing your changes will not resolve this problem.

bug#5803: 23.1.94; Using tq-enqueue in timer messes with keystroke echoing

[Stefan Monnier]

> Btw, does this mean the recipe Lars posted here in
>
>   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=5803#23
>
> works as expected in your patched Emacs, i.e. you do see the "C-x"
> echo after about a second?

Yes.

I pushed the patch.


        Stefan

bug#5803: 23.1.94; Using tq-enqueue in timer messes with keystroke echoing

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> works as expected in your patched Emacs, i.e. you do see the "C-x"
>> echo after about a second?
>
> Yes.
>
> I pushed the patch.

And it does indeed fix the test case here too, so I'm closing this bug
report.  (And Emacs also seems to work fine otherwise, which is always a
good sign. :-))

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#5803: 23.1.94; Using tq-enqueue in timer messes with keystroke echoing

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Since this patch mostly just removes code, I'll be delighted to install it.
>> Should I take this question as an actual "go ahead"?
>
> Let's wait for Lars to chime in, after he reads the past discussions
> on this.

I had a look at the thread and Stefan's patch (which removes a whole
bunch of code), but this is in an area that's mostly unfamiliar to me,
so I don't have much input here to give.  I'd say -- give it a go, and
see what happens.  We can always revert if it leads to problems.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#5803: 23.1.94; Using tq-enqueue in timer messes with keystroke echoing

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  5803@debbugs.gnu.org,
>   deng@randomsample.de
> Date: Fri, 11 Dec 2020 13:33:44 +0100
> 
> I had a look at the thread and Stefan's patch (which removes a whole
> bunch of code), but this is in an area that's mostly unfamiliar to me,
> so I don't have much input here to give.  I'd say -- give it a go, and
> see what happens.  We can always revert if it leads to problems.

No objection from me, assuming the problem is fixed by that patch.
Stefan?

bug#5803: Using tq-enqueue in timer messes with keystroke echoing

[Platon Pronko]

Hi!

It seems this patch did break something, see bug#45292 for example and analysis.

bug#5803: Using tq-enqueue in timer messes with keystroke echoing

[Platon Pronko]

Community member helpfully suggested that I didn't CC the relevant developers on my previous email, so the notification might be missed - doing so now (apologies in advance if I misunderstood the workflow and am spamming needlessly).

To recap: removal of record_asynch_buffer_change() at the end of read_and_dispose_of_process_output() seems to cause buffer change to not propagate in certain cases (details in bug#45292).