bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Spencer Baugh]

1. Under X11, with the GTK or Lucid toolkits:
   emacs -Q
2. Become owner of the clipboard selection by killing some text; the
   starting comments in the scratch buffer are a good candidate.
3. Immediately afterwards (i.e. without copy and pasting text in another
   window), run:
   (call-process "sleep" nil nil nil "inf")
4. Now other applications will hang when they attempt to paste text.
   Google Chrome and Slack are two examples.  (GTK-based applications
   seem to be fine.  So much for proprietary software...)

I guess Emacs doesn't handle requests for the X selection when in
call_process (which ultimately calls get_child_status).

It is regrettable that some other applications seem to not properly
handle a non-responsive X selection owner, but nevertheless ideally we'd
fix this in Emacs.


In GNU Emacs 29.0.90 (build 6, x86_64-pc-linux-gnu, GTK+ Version
 3.22.30, cairo version 1.15.12) of 2023-06-02 built on igm-qws-u22796a
Repository revision: ff6163fac51759945aa619ca6bf28413be4a53e0
Repository branch: emacs-29
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: CentOS Linux 7 (Core)

Configured using:
 'configure --with-x-toolkit=gtk --with-gif=ifavailable'

Configured features:
CAIRO DBUS FREETYPE GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LIBSELINUX LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND
SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS X11 XDBE XIM XINPUT2 XPM GTK3
ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date subr-x mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
cl-loaddefs cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils rmc iso-transl tooltip cconv eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify
dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit xinput2 x multi-tty make-network-process
emacs)

Memory information:
((conses 16 62487 9813)
 (symbols 48 9475 0)
 (strings 32 21897 2004)
 (string-bytes 1 660003)
 (vectors 16 9302)
 (vector-slots 8 148525 13706)
 (floats 8 32 21)
 (intervals 56 224 4)
 (buffers 976 10)
 (heap 1024 17392 1169))

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Date: Fri, 02 Jun 2023 21:55:09 -0400
> 
> 
> 1. Under X11, with the GTK or Lucid toolkits:
>    emacs -Q
> 2. Become owner of the clipboard selection by killing some text; the
>    starting comments in the scratch buffer are a good candidate.
> 3. Immediately afterwards (i.e. without copy and pasting text in another
>    window), run:
>    (call-process "sleep" nil nil nil "inf")
> 4. Now other applications will hang when they attempt to paste text.
>    Google Chrome and Slack are two examples.  (GTK-based applications
>    seem to be fine.  So much for proprietary software...)

Thanks.

Does this happen also with the latest pretest, v29.0.91?

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Date: Fri, 02 Jun 2023 21:55:09 -0400
>> 
>> 
>> 1. Under X11, with the GTK or Lucid toolkits:
>>    emacs -Q
>> 2. Become owner of the clipboard selection by killing some text; the
>>    starting comments in the scratch buffer are a good candidate.
>> 3. Immediately afterwards (i.e. without copy and pasting text in another
>>    window), run:
>>    (call-process "sleep" nil nil nil "inf")
>> 4. Now other applications will hang when they attempt to paste text.
>>    Google Chrome and Slack are two examples.  (GTK-based applications
>>    seem to be fine.  So much for proprietary software...)
>
> Thanks.
>
> Does this happen also with the latest pretest, v29.0.91?

I can't reproduce this, but the closest thing to Google Chrome on the
computer I am currently using is Firefox 10.0.7.  So would you also
build Emacs with -DTRACE_SELECTION, and show what is printed by Emacs
when the requestor hangs?

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Spencer Baugh <sbaugh@janestreet.com>,  63865@debbugs.gnu.org
> Date: Sat, 03 Jun 2023 15:11:19 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Spencer Baugh <sbaugh@janestreet.com>
> >> Date: Fri, 02 Jun 2023 21:55:09 -0400
> >> 
> >> 
> >> 1. Under X11, with the GTK or Lucid toolkits:
> >>    emacs -Q
> >> 2. Become owner of the clipboard selection by killing some text; the
> >>    starting comments in the scratch buffer are a good candidate.
> >> 3. Immediately afterwards (i.e. without copy and pasting text in another
> >>    window), run:
> >>    (call-process "sleep" nil nil nil "inf")
> >> 4. Now other applications will hang when they attempt to paste text.
> >>    Google Chrome and Slack are two examples.  (GTK-based applications
> >>    seem to be fine.  So much for proprietary software...)
> >
> > Thanks.
> >
> > Does this happen also with the latest pretest, v29.0.91?
> 
> I can't reproduce this, but the closest thing to Google Chrome on the
> computer I am currently using is Firefox 10.0.7.

It is strange that you cannot reproduce this.  I thought that when
another application requests a selection or clipboard text, the owner
(Emacs in this case) gets called, and since Emacs is sleeping, we
won't respond.

Are there some other factors involved in this interaction that I'm
missing?

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> It is strange that you cannot reproduce this.  I thought that when
> another application requests a selection or clipboard text, the owner
> (Emacs in this case) gets called, and since Emacs is sleeping, we
> won't respond.
>
> Are there some other factors involved in this interaction that I'm
> missing?

In this case, the requestor should time out and give up (which is what I
see.)  It should not simply freeze.  So my guess is Emacs is responding
to the selection request in some way that confuses the requestor.

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Spencer Baugh]

Po Lu <luangruo@yahoo.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Spencer Baugh <sbaugh@janestreet.com>
>>> Date: Fri, 02 Jun 2023 21:55:09 -0400
>>> 
>>> 
>>> 1. Under X11, with the GTK or Lucid toolkits:
>>>    emacs -Q
>>> 2. Become owner of the clipboard selection by killing some text; the
>>>    starting comments in the scratch buffer are a good candidate.
>>> 3. Immediately afterwards (i.e. without copy and pasting text in another
>>>    window), run:
>>>    (call-process "sleep" nil nil nil "inf")
>>> 4. Now other applications will hang when they attempt to paste text.
>>>    Google Chrome and Slack are two examples.  (GTK-based applications
>>>    seem to be fine.  So much for proprietary software...)
>>
>> Thanks.
>>
>> Does this happen also with the latest pretest, v29.0.91?
>
> I can't reproduce this, but the closest thing to Google Chrome on the
> computer I am currently using is Firefox 10.0.7.

BTW, this can also be reproduced using just Emacs.  If I try to paste in
another Emacs instead of in Google Chrome, I get a hang, followed by:

gui-get-selection: (error "Timed out waiting for reply from selection owner")

With -DTRACE_SELECTION on both Emacsen, the selection-owner Emacs prints
no logs, while the pasting Emacs prints the following log:

48866: Get selection UTF8_STRING, type _EMACS_TMP_
48866:   Start waiting 5 secs for SelectionNotify.
48866:     Waiting for 0 nsecs in addition.
48866:   Got event = NO
48866: Get selection TIMESTAMP, type _EMACS_TMP_
48866:   Start waiting 5 secs for SelectionNotify.
48866:     Waiting for 0 nsecs in addition.
48866:   Got event = NO

> So would you also build Emacs with -DTRACE_SELECTION, and show what is
> printed by Emacs when the requestor hangs?

Emacs prints nothing while stuck in call-process and the requestor
(Chrome) is hanging.  After I interrupt call-process, this log prints.

48290: x_handle_selection_event
48290: XGetAtomName --> NULL
48290: XGetAtomName --> text/plain;charset=utf-8
48290: x_handle_selection_request: selection=CLIPBOARD, target=text/plain;charset=utf-8
48290:  XInternAtom text/plain;charset=utf-8
48290:  x_start_selection_transfer: transferring to 0x1600000.  transfer consists of 70 bytes, quantum being 16777112
48290:  x_start_selection_transfer:  writing 70 elements directly to requestor window
48290: x_handle_selection_event
48290: x_handle_selection_request: selection=CLIPBOARD, target=TARGETS
48290:  x_start_selection_transfer: transferring to 0x1600000.  transfer consists of 92 bytes, quantum being 16777112
48290:  x_start_selection_transfer:  writing 23 elements directly to requestor window
48290: x_handle_selection_event
48290: x_handle_selection_request: selection=CLIPBOARD, target=TARGETS
48290:  x_start_selection_transfer: transferring to 0x1600000.  transfer consists of 92 bytes, quantum being 16777112
48290:  x_start_selection_transfer:  writing 23 elements directly to requestor window
48290: x_handle_selection_event
48290: x_handle_selection_request: selection=CLIPBOARD, target=UTF8_STRING
48290:  x_start_selection_transfer: transferring to 0x1600000.  transfer consists of 70 bytes, quantum being 16777112
48290:  x_start_selection_transfer:  writing 70 elements directly to requestor window
48290: x_handle_selection_event
48290: x_handle_selection_request: selection=CLIPBOARD, target=TARGETS
48290:  x_start_selection_transfer: transferring to 0x1600000.  transfer consists of 92 bytes, quantum being 16777112
48290:  x_start_selection_transfer:  writing 23 elements directly to requestor window
48290: x_handle_selection_event
48290: x_handle_selection_request: selection=CLIPBOARD, target=TARGETS
48290:  x_start_selection_transfer: transferring to 0x1600000.  transfer consists of 92 bytes, quantum being 16777112
48290:  x_start_selection_transfer:  writing 23 elements directly to requestor window
48290: x_handle_selection_event
48290: x_handle_selection_request: selection=CLIPBOARD, target=TARGETS
48290:  x_start_selection_transfer: transferring to 0x1600000.  transfer consists of 92 bytes, quantum being 16777112
48290:  x_start_selection_transfer:  writing 23 elements directly to requestor window
48290: x_handle_selection_event
48290: x_handle_selection_request: selection=CLIPBOARD, target=UTF8_STRING
48290:  x_start_selection_transfer: transferring to 0x1600000.  transfer consists of 70 bytes, quantum being 16777112
48290:  x_start_selection_transfer:  writing 70 elements directly to requestor window
48290: x_handle_selection_event
48290: x_handle_selection_request: selection=CLIPBOARD, target=TARGETS
48290:  x_start_selection_transfer: transferring to 0x1600000.  transfer consists of 92 bytes, quantum being 16777112
48290:  x_start_selection_transfer:  writing 23 elements directly to requestor window
48290: x_handle_selection_event
48290: x_handle_selection_request: selection=CLIPBOARD, target=TARGETS
48290:  x_start_selection_transfer: transferring to 0x1600000.  transfer consists of 92 bytes, quantum being 16777112
48290:  x_start_selection_transfer:  writing 23 elements directly to requestor window
48290: x_handle_selection_event
48290: x_handle_selection_request: selection=CLIPBOARD, target=TARGETS
48290:  x_start_selection_transfer: transferring to 0x1600000.  transfer consists of 92 bytes, quantum being 16777112
48290:  x_start_selection_transfer:  writing 23 elements directly to requestor window
48290: x_handle_selection_event
48290: x_handle_selection_request: selection=CLIPBOARD, target=UTF8_STRING
48290:  x_start_selection_transfer: transferring to 0x1600000.  transfer consists of 70 bytes, quantum being 16777112
48290:  x_start_selection_transfer:  writing 70 elements directly to requestor window
48290: x_handle_selection_event
48290: x_handle_selection_request: selection=CLIPBOARD, target=TARGETS
48290:  x_start_selection_transfer: transferring to 0x1600000.  transfer consists of 92 bytes, quantum being 16777112
48290:  x_start_selection_transfer:  writing 23 elements directly to requestor window
48290: x_handle_selection_event
48290: x_handle_selection_request: selection=CLIPBOARD, target=TARGETS
48290:  x_start_selection_transfer: transferring to 0x1600000.  transfer consists of 92 bytes, quantum being 16777112
48290:  x_start_selection_transfer:  writing 23 elements directly to requestor window
48290: x_handle_selection_event
48290: x_handle_selection_request: selection=CLIPBOARD, target=TARGETS
48290:  x_start_selection_transfer: transferring to 0x1600000.  transfer consists of 92 bytes, quantum being 16777112
48290:  x_start_selection_transfer:  writing 23 elements directly to requestor window
48290: x_handle_selection_event
48290: x_handle_selection_request: selection=CLIPBOARD, target=UTF8_STRING
48290:  x_start_selection_transfer: transferring to 0x1600000.  transfer consists of 70 bytes, quantum being 16777112
48290:  x_start_selection_transfer:  writing 70 elements directly to requestor window

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  63865@debbugs.gnu.org
> Date: Sat, 03 Jun 2023 07:10:41 -0400
> 
> BTW, this can also be reproduced using just Emacs.  If I try to paste in
> another Emacs instead of in Google Chrome, I get a hang, followed by:
> 
> gui-get-selection: (error "Timed out waiting for reply from selection owner")

Isn't this exactly what is expected to happen in this case?  Po Lu
said, in response to my question:

> > It is strange that you cannot reproduce this.  I thought that when
> > another application requests a selection or clipboard text, the owner
> > (Emacs in this case) gets called, and since Emacs is sleeping, we
> > won't respond.
> >
> > Are there some other factors involved in this interaction that I'm
> > missing?
> 
> In this case, the requestor should time out and give up (which is what I
> see.)  It should not simply freeze.

And that is exactly what happens when Emacs requests the selection: it
times out.  What else did you expect?

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Spencer Baugh <sbaugh@janestreet.com> writes:

> Po Lu <luangruo@yahoo.com> writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> From: Spencer Baugh <sbaugh@janestreet.com>
>>>> Date: Fri, 02 Jun 2023 21:55:09 -0400
>>>> 
>>>> 
>>>> 1. Under X11, with the GTK or Lucid toolkits:
>>>>    emacs -Q
>>>> 2. Become owner of the clipboard selection by killing some text; the
>>>>    starting comments in the scratch buffer are a good candidate.
>>>> 3. Immediately afterwards (i.e. without copy and pasting text in another
>>>>    window), run:
>>>>    (call-process "sleep" nil nil nil "inf")
>>>> 4. Now other applications will hang when they attempt to paste text.
>>>>    Google Chrome and Slack are two examples.  (GTK-based applications
>>>>    seem to be fine.  So much for proprietary software...)
>>>
>>> Thanks.
>>>
>>> Does this happen also with the latest pretest, v29.0.91?
>>
>> I can't reproduce this, but the closest thing to Google Chrome on the
>> computer I am currently using is Firefox 10.0.7.
>
> BTW, this can also be reproduced using just Emacs.  If I try to paste in
> another Emacs instead of in Google Chrome, I get a hang, followed by:
>
> gui-get-selection: (error "Timed out waiting for reply from selection owner")

Emacs doesn't hang...

> With -DTRACE_SELECTION on both Emacsen, the selection-owner Emacs prints
> no logs, while the pasting Emacs prints the following log:
>
> 48866: Get selection UTF8_STRING, type _EMACS_TMP_
> 48866:   Start waiting 5 secs for SelectionNotify.
> 48866:     Waiting for 0 nsecs in addition.
> 48866:   Got event = NO
> 48866: Get selection TIMESTAMP, type _EMACS_TMP_
> 48866:   Start waiting 5 secs for SelectionNotify.
> 48866:     Waiting for 0 nsecs in addition.
> 48866:   Got event = NO
>
>> So would you also build Emacs with -DTRACE_SELECTION, and show what is
>> printed by Emacs when the requestor hangs?
>
> Emacs prints nothing while stuck in call-process and the requestor
> (Chrome) is hanging.

OK, but then this is not a bug: Emacs can not respond to selection
requests when it is not reading keyboard input.

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Spencer Baugh]

Po Lu <luangruo@yahoo.com> writes:

> Spencer Baugh <sbaugh@janestreet.com> writes:
>
>> Po Lu <luangruo@yahoo.com> writes:
>>
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>
>>>>> From: Spencer Baugh <sbaugh@janestreet.com>
>>>>> Date: Fri, 02 Jun 2023 21:55:09 -0400
>>>>> 
>>>>> 
>>>>> 1. Under X11, with the GTK or Lucid toolkits:
>>>>>    emacs -Q
>>>>> 2. Become owner of the clipboard selection by killing some text; the
>>>>>    starting comments in the scratch buffer are a good candidate.
>>>>> 3. Immediately afterwards (i.e. without copy and pasting text in another
>>>>>    window), run:
>>>>>    (call-process "sleep" nil nil nil "inf")
>>>>> 4. Now other applications will hang when they attempt to paste text.
>>>>>    Google Chrome and Slack are two examples.  (GTK-based applications
>>>>>    seem to be fine.  So much for proprietary software...)
>>>>
>>>> Thanks.
>>>>
>>>> Does this happen also with the latest pretest, v29.0.91?
>>>
>>> I can't reproduce this, but the closest thing to Google Chrome on the
>>> computer I am currently using is Firefox 10.0.7.
>>
>> BTW, this can also be reproduced using just Emacs.  If I try to paste in
>> another Emacs instead of in Google Chrome, I get a hang, followed by:
>>
>> gui-get-selection: (error "Timed out waiting for reply from selection owner")
>
> Emacs doesn't hang...

Well, it hangs for 10 seconds, then times out, because Emacs is
correctly implemented.  Some other applications are incorrectly
implemented and never time out.

>> With -DTRACE_SELECTION on both Emacsen, the selection-owner Emacs prints
>> no logs, while the pasting Emacs prints the following log:
>>
>> 48866: Get selection UTF8_STRING, type _EMACS_TMP_
>> 48866:   Start waiting 5 secs for SelectionNotify.
>> 48866:     Waiting for 0 nsecs in addition.
>> 48866:   Got event = NO
>> 48866: Get selection TIMESTAMP, type _EMACS_TMP_
>> 48866:   Start waiting 5 secs for SelectionNotify.
>> 48866:     Waiting for 0 nsecs in addition.
>> 48866:   Got event = NO
>>
>>> So would you also build Emacs with -DTRACE_SELECTION, and show what is
>>> printed by Emacs when the requestor hangs?
>>
>> Emacs prints nothing while stuck in call-process and the requestor
>> (Chrome) is hanging.
>
> OK, but then this is not a bug: Emacs can not respond to selection
> requests when it is not reading keyboard input.

"Emacs can not respond to selection requests when it is not reading
keyboard input." sounds like a bug to me!  Even if it's hard to fix,
it's still a bug.

If I'm implementing some package and I decide to use call-process for
some long operation, then some user uses my package and it runs
call-process, and they get bored while waiting and switch away from
Emacs, they'll experience a hang in some other application.  That hang
seems clearly undesirable!

(I should mention that Slack (and possibly all Electron applications)
have worse behavior than I originally said: They don't just hang on
paste, their UI hangs completely while Emacs is in call-process.  (I
guess these applications are constantly requesting the selection or
something?)  Which is an extremely bad experience for the users of
packages which use call-process...)

I'm personally working around this by replacing call-process with
start-process and accept-process-output.  Because otherwise my packages
(and any other package using call-process ever) will cause random hangs
in other applications, which is obviously bad and not something anyone
would want.

So perhaps call-process on Unix should be reimplemented in terms of
those functions?  Or if that would change behavior too much, perhaps
call-process should be deprecated in favor of some new helper built on
those?

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  63865@debbugs.gnu.org
> Date: Sat, 03 Jun 2023 08:30:57 -0400
> 
> Po Lu <luangruo@yahoo.com> writes:
> 
> > Spencer Baugh <sbaugh@janestreet.com> writes:
> >
> >> Po Lu <luangruo@yahoo.com> writes:
> >>
> >>> Eli Zaretskii <eliz@gnu.org> writes:
> >>>
> >>>>> From: Spencer Baugh <sbaugh@janestreet.com>
> >>>>> Date: Fri, 02 Jun 2023 21:55:09 -0400
> >>>>> 
> >>>>> 
> >>>>> 1. Under X11, with the GTK or Lucid toolkits:
> >>>>>    emacs -Q
> >>>>> 2. Become owner of the clipboard selection by killing some text; the
> >>>>>    starting comments in the scratch buffer are a good candidate.
> >>>>> 3. Immediately afterwards (i.e. without copy and pasting text in another
> >>>>>    window), run:
> >>>>>    (call-process "sleep" nil nil nil "inf")
> >>>>> 4. Now other applications will hang when they attempt to paste text.
> >>>>>    Google Chrome and Slack are two examples.  (GTK-based applications
> >>>>>    seem to be fine.  So much for proprietary software...)
> >>>>
> >>>> Thanks.
> >>>>
> >>>> Does this happen also with the latest pretest, v29.0.91?
> >>>
> >>> I can't reproduce this, but the closest thing to Google Chrome on the
> >>> computer I am currently using is Firefox 10.0.7.
> >>
> >> BTW, this can also be reproduced using just Emacs.  If I try to paste in
> >> another Emacs instead of in Google Chrome, I get a hang, followed by:
> >>
> >> gui-get-selection: (error "Timed out waiting for reply from selection owner")
> >
> > Emacs doesn't hang...
> 
> Well, it hangs for 10 seconds, then times out, because Emacs is
> correctly implemented.

The value of the timeout can be customized, if you don't like the
default.

> > OK, but then this is not a bug: Emacs can not respond to selection
> > requests when it is not reading keyboard input.
> 
> "Emacs can not respond to selection requests when it is not reading
> keyboard input." sounds like a bug to me!  Even if it's hard to fix,
> it's still a bug.

We run Lisp to generate selection response, so there's no way we can
do that when the main Lisp thread is busy.  It isn't a bug, it's a
restriction of how Emacs is designed.

> If I'm implementing some package and I decide to use call-process for
> some long operation, then some user uses my package and it runs
> call-process, and they get bored while waiting and switch away from
> Emacs, they'll experience a hang in some other application.  That hang
> seems clearly undesirable!

Then don't design the package such that call-process blocks Emacs for
prolonged periods of time.  Because this will annoy the users of Emacs
even before it will be seen by other applications that request X
selections.

> I'm personally working around this by replacing call-process with
> start-process and accept-process-output.  Because otherwise my packages
> (and any other package using call-process ever) will cause random hangs
> in other applications, which is obviously bad and not something anyone
> would want.
> 
> So perhaps call-process on Unix should be reimplemented in terms of
> those functions?  Or if that would change behavior too much, perhaps
> call-process should be deprecated in favor of some new helper built on
> those?

call-process has its use cases, which are important, and we will not
deprecate it.

You can easily emulate call-process with start-process if you need to
do so, so Emacs gives you both possibilities (and expects you to use
whatever is right in each case).

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Spencer Baugh]

Eli Zaretskii <eliz@gnu.org> writes:
>> If I'm implementing some package and I decide to use call-process for
>> some long operation, then some user uses my package and it runs
>> call-process, and they get bored while waiting and switch away from
>> Emacs, they'll experience a hang in some other application.  That hang
>> seems clearly undesirable!
>
> Then don't design the package such that call-process blocks Emacs for
> prolonged periods of time.  Because this will annoy the users of Emacs
> even before it will be seen by other applications that request X
> selections.

Forget other packages: Emacs itself uses call-process in tons of places
where it will run for prolonged periods of time!

In fact, I just ran "C-x p g call-process" to search for instances, only
to find that that command itself uses call-process and hung my other
applications!

Should we port all these instances away from using call-process to avoid
this behavior?  I certainly would like to, because I don't like that
when I do a particularly long process-find-regexp or shell-command or
any other operation which uses call-process, it doesn't just hang Emacs,
it hangs basically my whole OS.

>> I'm personally working around this by replacing call-process with
>> start-process and accept-process-output.  Because otherwise my packages
>> (and any other package using call-process ever) will cause random hangs
>> in other applications, which is obviously bad and not something anyone
>> would want.
>> 
>> So perhaps call-process on Unix should be reimplemented in terms of
>> those functions?  Or if that would change behavior too much, perhaps
>> call-process should be deprecated in favor of some new helper built on
>> those?
>
> call-process has its use cases, which are important, and we will not
> deprecate it.
>
> You can easily emulate call-process with start-process if you need to
> do so, so Emacs gives you both possibilities (and expects you to use
> whatever is right in each case).

What use case does call-process have on Unix, which an emulation in
terms of start-process would not also satisfy?

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: luangruo@yahoo.com,  63865@debbugs.gnu.org
> Date: Sat, 03 Jun 2023 09:10:02 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> If I'm implementing some package and I decide to use call-process for
> >> some long operation, then some user uses my package and it runs
> >> call-process, and they get bored while waiting and switch away from
> >> Emacs, they'll experience a hang in some other application.  That hang
> >> seems clearly undesirable!
> >
> > Then don't design the package such that call-process blocks Emacs for
> > prolonged periods of time.  Because this will annoy the users of Emacs
> > even before it will be seen by other applications that request X
> > selections.
> 
> Forget other packages: Emacs itself uses call-process in tons of places
> where it will run for prolonged periods of time!

Really?  "tons of places where it will run for prolonged periods of
time"? what is "prolonged period of time" for this purpose?  Aren't
you a tad exaggerating?

> Should we port all these instances away from using call-process to avoid
> this behavior?

There's no general answer to that, we should examine each case
separately.

> > call-process has its use cases, which are important, and we will not
> > deprecate it.
> >
> > You can easily emulate call-process with start-process if you need to
> > do so, so Emacs gives you both possibilities (and expects you to use
> > whatever is right in each case).
> 
> What use case does call-process have on Unix, which an emulation in
> terms of start-process would not also satisfy?

When the process returns quickly.  call-process is significantly
simpler to use than start-process+wait, so doing that when unnecessary
is a complication we shouldn't take.

Anyway, this kind of discussion doesn't belong in a bug report about X
selections.

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Spencer Baugh]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Cc: luangruo@yahoo.com,  63865@debbugs.gnu.org
>> Date: Sat, 03 Jun 2023 09:10:02 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> >> If I'm implementing some package and I decide to use call-process for
>> >> some long operation, then some user uses my package and it runs
>> >> call-process, and they get bored while waiting and switch away from
>> >> Emacs, they'll experience a hang in some other application.  That hang
>> >> seems clearly undesirable!
>> >
>> > Then don't design the package such that call-process blocks Emacs for
>> > prolonged periods of time.  Because this will annoy the users of Emacs
>> > even before it will be seen by other applications that request X
>> > selections.
>> 
>> Forget other packages: Emacs itself uses call-process in tons of places
>> where it will run for prolonged periods of time!
>
> Really?  "tons of places where it will run for prolonged periods of
> time"? what is "prolonged period of time" for this purpose?  Aren't
> you a tad exaggerating?

"prolonged period of time" is anything over a second.

I see tons of calls to call-process with potentially long-running
programs.  "find", "gcc", "grep", "awk"... and depending on the user's
system, *any* subprocess call can potentially run for a long time.

But again, the point isn't just that they can potentially run for a long
time.  It's that *the user's whole system can be unusable* while they
run.  We are not just blocking Emacs, we are (sometimes) blocking
*everything*.

>> Should we port all these instances away from using call-process to avoid
>> this behavior?
>
> There's no general answer to that, we should examine each case
> separately.

A general answer could be fixing call-process to not hang other
processes.

>> > call-process has its use cases, which are important, and we will not
>> > deprecate it.
>> >
>> > You can easily emulate call-process with start-process if you need to
>> > do so, so Emacs gives you both possibilities (and expects you to use
>> > whatever is right in each case).
>> 
>> What use case does call-process have on Unix, which an emulation in
>> terms of start-process would not also satisfy?
>
> When the process returns quickly.  call-process is significantly
> simpler to use than start-process+wait, so doing that when unnecessary
> is a complication we shouldn't take.

What I mean was an emulation like this:

(defun my-call-process (program &optional destination &rest args)
  (let ((process (make-process :name "call-process"
                               :buffer destination
                               :command (cons program args))))
    (while (accept-process-output process))))

my-call-process is missing a few arguments that the real call-process
has.  But if this is all I use, is there *any* reason to *not* use
my-call-process on Linux?

There is at least one strong reason to use it: It won't hang other
processes.

> Anyway, this kind of discussion doesn't belong in a bug report about X
> selections.

But the entire point of this discussion, for me, is to fix the
X-selection-related hangs which Emacs currently causes when in
call-process.  i.e., this bug report.

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: luangruo@yahoo.com,  63865@debbugs.gnu.org
> Date: Sat, 03 Jun 2023 10:12:41 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Really?  "tons of places where it will run for prolonged periods of
> > time"? what is "prolonged period of time" for this purpose?  Aren't
> > you a tad exaggerating?
> 
> "prolonged period of time" is anything over a second.
> 
> I see tons of calls to call-process with potentially long-running
> programs.  "find", "gcc", "grep", "awk"... and depending on the user's
> system, *any* subprocess call can potentially run for a long time.

Please be specific.  For example, "grep" runs in an async subprocess;
we run it with call-process when we probe it for support of some
command-line option.  If you can show that these calls take "prolonged
period of time", please describe what you do and show the timing.

> But again, the point isn't just that they can potentially run for a long
> time.  It's that *the user's whole system can be unusable* while they
> run.  We are not just blocking Emacs, we are (sometimes) blocking
> *everything*.

AFAIU, the only "everything" that is blocked is an application that
happens to request an X selection at that precise time.  That should
be rare for short enough call-process runs, since the same user is
doing that.

> (defun my-call-process (program &optional destination &rest args)
>   (let ((process (make-process :name "call-process"
>                                :buffer destination
>                                :command (cons program args))))
>     (while (accept-process-output process))))
> 
> my-call-process is missing a few arguments that the real call-process
> has.  But if this is all I use, is there *any* reason to *not* use
> my-call-process on Linux?

You are free to do it if it suits your needs.  Emacs provides those
primitives to you.

But saying that this should replace call-process in each and every
case is a non-starter.

> There is at least one strong reason to use it: It won't hang other
> processes.

Yes, and gazillion complications, like handling the SIGCHLD signal,
pselect interface, etc.

> > Anyway, this kind of discussion doesn't belong in a bug report about X
> > selections.
> 
> But the entire point of this discussion, for me, is to fix the
> X-selection-related hangs which Emacs currently causes when in
> call-process.  i.e., this bug report.

It doesn't matter, because you insist on making the issue much more
general.

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Spencer Baugh]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Cc: luangruo@yahoo.com,  63865@debbugs.gnu.org
>> Date: Sat, 03 Jun 2023 10:12:41 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Really?  "tons of places where it will run for prolonged periods of
>> > time"? what is "prolonged period of time" for this purpose?  Aren't
>> > you a tad exaggerating?
>> 
>> "prolonged period of time" is anything over a second.
>> 
>> I see tons of calls to call-process with potentially long-running
>> programs.  "find", "gcc", "grep", "awk"... and depending on the user's
>> system, *any* subprocess call can potentially run for a long time.
>
> Please be specific.  For example, "grep" runs in an async subprocess;
> we run it with call-process when we probe it for support of some
> command-line option.  If you can show that these calls take "prolonged
> period of time", please describe what you do and show the timing.

No, I mean "grep" called specifically by call-process.  As, for example,
I see happening in mh-grep-execute-search and nnspool-find-id.  In the
former case, it's a recursive grep which could take many seconds.

Or as I already mentioned: xref-matches-in-files (used by
project-find-regexp, C-x p g) uses call-process-region to run recursive
grep in an arbitrary directory.  Which could easily take minutes.

These are understandable designs, and I'm not too bothered by the fact
that they block Emacs.  But this becomes much worse when they are also
blocking other applications because of this X selection bug.

>> But again, the point isn't just that they can potentially run for a long
>> time.  It's that *the user's whole system can be unusable* while they
>> run.  We are not just blocking Emacs, we are (sometimes) blocking
>> *everything*.
>
> AFAIU, the only "everything" that is blocked is an application that
> happens to request an X selection at that precise time.  That should
> be rare for short enough call-process runs, since the same user is
> doing that.

No, as I mentioned earlier, some (buggy, of course) applications
frequently request the X selection; such as Slack (and possibly all
other Electron applications too).  Such applications just immediately
hang during the entire call-process run without any user interaction.

In my case, Google Chrome and Slack are the only non-Emacs applications
I use.  So... for me, everything gets blocked.

>> (defun my-call-process (program &optional destination &rest args)
>>   (let ((process (make-process :name "call-process"
>>                                :buffer destination
>>                                :command (cons program args))))
>>     (while (accept-process-output process))))
>> 
>> my-call-process is missing a few arguments that the real call-process
>> has.  But if this is all I use, is there *any* reason to *not* use
>> my-call-process on Linux?
>
> You are free to do it if it suits your needs.  Emacs provides those
> primitives to you.
>
> But saying that this should replace call-process in each and every
> case is a non-starter.
>
>> There is at least one strong reason to use it: It won't hang other
>> processes.
>
> Yes, and gazillion complications, like handling the SIGCHLD signal,
> pselect interface, etc.
>
>> > Anyway, this kind of discussion doesn't belong in a bug report about X
>> > selections.
>> 
>> But the entire point of this discussion, for me, is to fix the
>> X-selection-related hangs which Emacs currently causes when in
>> call-process.  i.e., this bug report.
>
> It doesn't matter, because you insist on making the issue much more
> general.

I would be happy with a targeted, specific fix for the bad behavior I
reported.

Here's a specific instance that would be good to fix: If I run "M-!
sleep 30 RET", that will cause some applications to hang while Emacs is
waiting on the sleep; sometimes (as with Slack) without user
interaction, or sometimes only if the user tries to paste in them.  Do
you have a suggestion on how to fix that?

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: luangruo@yahoo.com,  63865@debbugs.gnu.org
> Date: Sat, 03 Jun 2023 10:41:15 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Please be specific.  For example, "grep" runs in an async subprocess;
> > we run it with call-process when we probe it for support of some
> > command-line option.  If you can show that these calls take "prolonged
> > period of time", please describe what you do and show the timing.
> 
> No, I mean "grep" called specifically by call-process.  As, for example,
> I see happening in mh-grep-execute-search and nnspool-find-id.  In the
> former case, it's a recursive grep which could take many seconds.
>
> Or as I already mentioned: xref-matches-in-files (used by
> project-find-regexp, C-x p g) uses call-process-region to run recursive
> grep in an arbitrary directory.  Which could easily take minutes.

I hope the respective developers will consider whether it is
reasonably feasible to make those features non-blocking.

> These are understandable designs, and I'm not too bothered by the fact
> that they block Emacs.  But this becomes much worse when they are also
> blocking other applications because of this X selection bug.

Only the buggy ones, let's not forget.  The non-buggy ones could wait
without blocking, just like you expect Emacs to do.

> > AFAIU, the only "everything" that is blocked is an application that
> > happens to request an X selection at that precise time.  That should
> > be rare for short enough call-process runs, since the same user is
> > doing that.
> 
> No, as I mentioned earlier, some (buggy, of course) applications
> frequently request the X selection; such as Slack (and possibly all
> other Electron applications too).  Such applications just immediately
> hang during the entire call-process run without any user interaction.

It is unreasonable to expect Emacs to solve bugs in other
applications.  We are having hard time to solve our own.

> I would be happy with a targeted, specific fix for the bad behavior I
> reported.
> 
> Here's a specific instance that would be good to fix: If I run "M-!
> sleep 30 RET", that will cause some applications to hang while Emacs is
> waiting on the sleep; sometimes (as with Slack) without user
> interaction, or sometimes only if the user tries to paste in them.  Do
> you have a suggestion on how to fix that?

No, I don't.  And I explained why at the very beginning.  I invite you
to read xselect.c and see what kind of processing we do there to
handle selection requests.

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Spencer Baugh]

Eli Zaretskii <eliz@gnu.org> writes:
>> I would be happy with a targeted, specific fix for the bad behavior I
>> reported.
>> 
>> Here's a specific instance that would be good to fix: If I run "M-!
>> sleep 30 RET", that will cause some applications to hang while Emacs is
>> waiting on the sleep; sometimes (as with Slack) without user
>> interaction, or sometimes only if the user tries to paste in them.  Do
>> you have a suggestion on how to fix that?
>
> No, I don't.  And I explained why at the very beginning.  I invite you
> to read xselect.c and see what kind of processing we do there to
> handle selection requests.

What about a new version of call-process, maybe
"call-process-allow-lisp", which doesn't stop timers/process
filters/Lisp/etc from running while Emacs is blocked in it?  Then any
caller which doesn't want to stop Lisp from running while they are
waiting for the subprocess can use "call-process-allow-lisp" instead of
"call-process".

I can implement that if it sounds desirable.  I can also send a mail to
emacs-devel if you want more discussion of it beforehand.

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: luangruo@yahoo.com,  63865@debbugs.gnu.org
> Date: Sat, 03 Jun 2023 11:34:44 -0400
> 
> What about a new version of call-process, maybe
> "call-process-allow-lisp", which doesn't stop timers/process
> filters/Lisp/etc from running while Emacs is blocked in it?  Then any
> caller which doesn't want to stop Lisp from running while they are
> waiting for the subprocess can use "call-process-allow-lisp" instead of
> "call-process".

If "Emacs is blocked", timers and process filters cannot run.  So I'm
not sure I understand what you mean here.

if you mean to use start-process instead, then why exactly do we need
a new function?  The building blocks for coding that are already
available, and I very much doubt that you will be able to come up with
one-fits-all combination of them anyway, because of the different
needs: what exactly should be blocked and what not, and how.

> I can implement that if it sounds desirable.  I can also send a mail to
> emacs-devel if you want more discussion of it beforehand.

Yes, please.  This discussion doesn't belong here.

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Andreas Schwab]

On Jun 03 2023, Spencer Baugh wrote:

> Forget other packages: Emacs itself uses call-process in tons of places
> where it will run for prolonged periods of time!

This has nothing to do with call-process.  Emacs can be busy for many
reasons.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Spencer Baugh]

Andreas Schwab <schwab@linux-m68k.org> writes:
> On Jun 03 2023, Spencer Baugh wrote:
>> Forget other packages: Emacs itself uses call-process in tons of places
>> where it will run for prolonged periods of time!
>
> This has nothing to do with call-process.  Emacs can be busy for many
> reasons.

Other things which make Emacs busy still let Emacs respond to X
selection requests.

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Cc: luangruo@yahoo.com,  Eli Zaretskii <eliz@gnu.org>,  63865@debbugs.gnu.org
> Date: Sat, 03 Jun 2023 13:03:14 -0400
> 
> Andreas Schwab <schwab@linux-m68k.org> writes:
> > On Jun 03 2023, Spencer Baugh wrote:
> >> Forget other packages: Emacs itself uses call-process in tons of places
> >> where it will run for prolonged periods of time!
> >
> > This has nothing to do with call-process.  Emacs can be busy for many
> > reasons.
> 
> Other things which make Emacs busy still let Emacs respond to X
> selection requests.

Which "other things"?

Andreas is right: anything that the Lisp thread does prevents it from
responding to X selection requests, because while a Lisp program runs,
Emacs checks the input queue only occasionally, as part of maybe_quit
processing.  So when the main thread is busy by doing anything, all
bets are off.

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Spencer Baugh <sbaugh@janestreet.com> writes:

> "Emacs can not respond to selection requests when it is not reading
> keyboard input." sounds like a bug to me!  Even if it's hard to fix,
> it's still a bug.
>
> If I'm implementing some package and I decide to use call-process for
> some long operation, then some user uses my package and it runs
> call-process, and they get bored while waiting and switch away from
> Emacs, they'll experience a hang in some other application.  That hang
> seems clearly undesirable!
>
> (I should mention that Slack (and possibly all Electron applications)
> have worse behavior than I originally said: They don't just hang on
> paste, their UI hangs completely while Emacs is in call-process.  (I
> guess these applications are constantly requesting the selection or
> something?)  Which is an extremely bad experience for the users of
> packages which use call-process...)
>
> I'm personally working around this by replacing call-process with
> start-process and accept-process-output.  Because otherwise my packages
> (and any other package using call-process ever) will cause random hangs
> in other applications, which is obviously bad and not something anyone
> would want.
>
> So perhaps call-process on Unix should be reimplemented in terms of
> those functions?  Or if that would change behavior too much, perhaps
> call-process should be deprecated in favor of some new helper built on
> those?

Do you expect completely arbitrary Lisp code (which may perform _any_
operations, including reading user input) to run inside call-process?
Because Emacs runs such code upon receiving a selection request.

Hangs in badly implemented programs are a small price to pay for such
flexibility.  Try getting a clipboard manager: it may resolve the issue.

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Spencer Baugh <sbaugh@janestreet.com> writes:

> What use case does call-process have on Unix, which an emulation in
> terms of start-process would not also satisfy?

That random Lisp does not run inside.

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Spencer Baugh <sbaugh@janestreet.com> writes:

> Other things which make Emacs busy still let Emacs respond to X
> selection requests.

Only if Emacs is still reading keyboard input.  Responding to selection
requests is only possible when it is safe to run Lisp.  A cursory
examination of xselect.el will show why.

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> AFAIU, the only "everything" that is blocked is an application that
> happens to request an X selection at that precise time.  That should
> be rare for short enough call-process runs, since the same user is
> doing that.

Btw, it seems only fair to mention that hangs here are clearly the fault
of the program that hangs.  All reasonably implemented X clients
implement a time-out mechanism and listen for the selection owner being
deleted.

bug#63865: 29.0.90; call-process while owning the X selection hangs other processes

[Spencer Baugh]

BTW, the buggy behavior in Slack at least can be fixed by running:

slack 'slack://setting/?update={"isClipboardReadDisabled":true}'

then Slack won't constantly try to read the clipboard in the background.
(Apparently this may be fixed in the future, future readers should check
with Slack)

Though, I'm still of the opinion that we should make a
call-process-with-lisp which doesn't stop Lisp from running while
call-process-with-lisp is running, and use that everywhere.  Then we
wouldn't be triggering such bugs...