bug#25214: 26.0.50; Interacting with user from threads other than the primary

bug#25214: 26.0.50; Interacting with user from threads other than the primary

[Eli Zaretskii]

Here's some fun with threads:

 emacs -Q

Evaluate:

  (defun my-question ()
    (let (use-dialog-box) ;; in case this is a GUI frame
      (if (y-or-n-p "Yes or no? ")
	  (message "You said YES")
	(message "You said NO"))))

Evaluate:

  (my-question)

You get a question asked in the minibuffer, answer it with Y or N, and
get the corresponding echo.  All's well.

Now evaluate this:

  (make-thread #'my-question "question")

You again get the question asked in the minibuffer, but typing
anything at the prompt causes Emacs to complain that whatever you
typed is "undefined".  Your only fire escape is to close this session
with some mouse gesture, or kill it.

What happens here is that, by the time the new thread runs, the main
(a.k.a. "primary") thread is already idle, i.e. it already called
read_char, which called wait_reading_process_output, which called
thread_select.  (In fact, it's _because_ the primary thread called
thread_select that the new thread was allowed to run at all, since it
has to acquire the global lock for that, and that is only possible
when the running thread releases the lock inside thread_select.)  Now,
when wait_reading_process_output calls thread_select, it prepares the
mask for file descriptors it will wait for to become ready for
reading, in this case there's only one descriptor: the keyboard
descriptor.  So the primary thread is waiting for input from the
keyboard.

Now the new thread starts running and eventually calls y-or-n-p, which
calls read_char, which calls wait_reading_process_output.  But when
this call prepares the mask for the thread_select call, it skips the
keyboard descriptor, because we only allow a single thread to wait on
each descriptor, and the keyboard descriptor is already watched by the
primary thread.  So the new thread ends up not waiting on the keyboard
input descriptor.

The thread_select call then switches back to the primary thread, and
the primary thread receives the Y or N character you type.  But it
doesn't know what to do with it, so it becomes confused.

IOW, user interaction from non-primary threads is currently inherently
unsafe.

And then, of course, there's the use case where two threads ask the
user something via the minibuffer.

Thoughts?



In GNU Emacs 26.0.50.118 (i686-pc-mingw32)
 of 2016-12-16 built on HOME-C4E4A596F7
Repository revision: fb2fdb1435d2520c1cbf2a3d6a53128512a38458
Windowing system distributor 'Microsoft Corp.', version 5.1.2600
Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Configured using:
 'configure --prefix=/d/usr --enable-checking=yes,glyphs --with-wide-int
 --with-modules 'CFLAGS=-O0 -gdwarf-4 -g3''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS MODULES

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1255

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message subr-x puny seq byte-opt gv
bytecomp byte-compile cl-extra help-mode cconv cl-loaddefs pcase cl-lib
dired dired-loaddefs format-spec rfc822 mml easymenu mml-sec
password-cache epa derived epg epg-config gnus-util rmail rmail-loaddefs
mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils
mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr
mail-utils time-date mule-util tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel dos-w32 ls-lisp disp-table
term/w32-win w32-win w32-vars term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript case-table epa-hook jka-cmpr-hook help
simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs button
faces cus-face macroexp files text-properties overlay sha1 md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote w32notify w32 multi-tty make-network-process emacs)

Memory information:
((conses 16 102990 12105)
 (symbols 56 21101 0)
 (miscs 48 36 129)
 (strings 16 21030 4737)
 (string-bytes 1 593607)
 (vectors 16 14398)
 (vector-slots 8 444812 4412)
 (floats 8 179 129)
 (intervals 40 269 104)
 (buffers 864 11))

bug#25214: 26.0.50; Interacting with user from threads other than the primary

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 6740 bytes --]

Hi Eli,

> Thoughts?

I don't have that much concurrency experience, but this sounds like an issue that other languages have had to deal with in the context of GUIs.  In Microsoft's Windows Forms, for example, applications start with a single thread (the UI thread) running a message pump, and other threads that want to interact with the user may do so only by posting a message (using Control.Invoke) to the UI thread.

If I understand your description of the problem correctly, what happens is essentially that the main thread and the background thread are competing for keyboard access.  If so, could the solution be to route all UI requests through the main thread?  Concretely, this would mean that in the context of a secondary thread, calling "read-from-minibuffer" really means pushing a message (tagged with a thread id) into a queue of "pending-reads-from-minibuffer", then repeatedly `thread-yield`ing until the UI thread has pushed the user's response into a map of "processed-reads-from-minibuffer".  The user's response would be collected by the main thread.

Alternatively, could we simply disallow this type of interaction until we have a good story about it? That would bring us closer to the way web workers work in Javascript (they need to post a message to interact with the UI/DOM).

Cheers,
Clément.

On 2016-12-16 10:17, Eli Zaretskii wrote:
> Here's some fun with threads:
> 
>  emacs -Q
> 
> Evaluate:
> 
>   (defun my-question ()
>     (let (use-dialog-box) ;; in case this is a GUI frame
>       (if (y-or-n-p "Yes or no? ")
> 	  (message "You said YES")
> 	(message "You said NO"))))
> 
> Evaluate:
> 
>   (my-question)
> 
> You get a question asked in the minibuffer, answer it with Y or N, and
> get the corresponding echo.  All's well.
> 
> Now evaluate this:
> 
>   (make-thread #'my-question "question")
> 
> You again get the question asked in the minibuffer, but typing
> anything at the prompt causes Emacs to complain that whatever you
> typed is "undefined".  Your only fire escape is to close this session
> with some mouse gesture, or kill it.
> 
> What happens here is that, by the time the new thread runs, the main
> (a.k.a. "primary") thread is already idle, i.e. it already called
> read_char, which called wait_reading_process_output, which called
> thread_select.  (In fact, it's _because_ the primary thread called
> thread_select that the new thread was allowed to run at all, since it
> has to acquire the global lock for that, and that is only possible
> when the running thread releases the lock inside thread_select.)  Now,
> when wait_reading_process_output calls thread_select, it prepares the
> mask for file descriptors it will wait for to become ready for
> reading, in this case there's only one descriptor: the keyboard
> descriptor.  So the primary thread is waiting for input from the
> keyboard.
> 
> Now the new thread starts running and eventually calls y-or-n-p, which
> calls read_char, which calls wait_reading_process_output.  But when
> this call prepares the mask for the thread_select call, it skips the
> keyboard descriptor, because we only allow a single thread to wait on
> each descriptor, and the keyboard descriptor is already watched by the
> primary thread.  So the new thread ends up not waiting on the keyboard
> input descriptor.
> 
> The thread_select call then switches back to the primary thread, and
> the primary thread receives the Y or N character you type.  But it
> doesn't know what to do with it, so it becomes confused.
> 
> IOW, user interaction from non-primary threads is currently inherently
> unsafe.
> 
> And then, of course, there's the use case where two threads ask the
> user something via the minibuffer.
> 
> Thoughts?
> 
> 
> 
> In GNU Emacs 26.0.50.118 (i686-pc-mingw32)
>  of 2016-12-16 built on HOME-C4E4A596F7
> Repository revision: fb2fdb1435d2520c1cbf2a3d6a53128512a38458
> Windowing system distributor 'Microsoft Corp.', version 5.1.2600
> Recent messages:
> For information about GNU Emacs and the GNU system, type C-h C-a.
> 
> Configured using:
>  'configure --prefix=/d/usr --enable-checking=yes,glyphs --with-wide-int
>  --with-modules 'CFLAGS=-O0 -gdwarf-4 -g3''
> 
> Configured features:
> XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY ACL GNUTLS LIBXML2 ZLIB
> TOOLKIT_SCROLL_BARS MODULES
> 
> Important settings:
>   value of $LANG: ENU
>   locale-coding-system: cp1255
> 
> Major mode: Lisp Interaction
> 
> Minor modes in effect:
>   tooltip-mode: t
>   global-eldoc-mode: t
>   electric-indent-mode: t
>   mouse-wheel-mode: t
>   tool-bar-mode: t
>   menu-bar-mode: t
>   file-name-shadow-mode: t
>   global-font-lock-mode: t
>   font-lock-mode: t
>   blink-cursor-mode: t
>   auto-composition-mode: t
>   auto-encryption-mode: t
>   auto-compression-mode: t
>   line-number-mode: t
>   transient-mark-mode: t
> 
> Load-path shadows:
> None found.
> 
> Features:
> (shadow sort mail-extr emacsbug message subr-x puny seq byte-opt gv
> bytecomp byte-compile cl-extra help-mode cconv cl-loaddefs pcase cl-lib
> dired dired-loaddefs format-spec rfc822 mml easymenu mml-sec
> password-cache epa derived epg epg-config gnus-util rmail rmail-loaddefs
> mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils
> mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr
> mail-utils time-date mule-util tooltip eldoc electric uniquify
> ediff-hook vc-hooks lisp-float-type mwheel dos-w32 ls-lisp disp-table
> term/w32-win w32-win w32-vars term/common-win tool-bar dnd fontset image
> regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
> lisp-mode prog-mode register page menu-bar rfn-eshadow isearch timer
> select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
> term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
> vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
> hebrew greek romanian slovak czech european ethiopic indian cyrillic
> chinese composite charscript case-table epa-hook jka-cmpr-hook help
> simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs button
> faces cus-face macroexp files text-properties overlay sha1 md5 base64
> format env code-pages mule custom widget hashtable-print-readable
> backquote w32notify w32 multi-tty make-network-process emacs)
> 
> Memory information:
> ((conses 16 102990 12105)
>  (symbols 56 21101 0)
>  (miscs 48 36 129)
>  (strings 16 21030 4737)
>  (string-bytes 1 593607)
>  (vectors 16 14398)
>  (vector-slots 8 444812 4412)
>  (floats 8 179 129)
>  (intervals 40 269 104)
>  (buffers 864 11))
> 
> 
> 
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

bug#25214: 26.0.50; Interacting with user from threads other than the primary

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

> Here's some fun with threads:
>
>  emacs -Q
>
> Evaluate:
>
>   (defun my-question ()
>     (let (use-dialog-box) ;; in case this is a GUI frame
>       (if (y-or-n-p "Yes or no? ")
> 	  (message "You said YES")
> 	(message "You said NO"))))
>
> Now evaluate this:
>
>   (make-thread #'my-question "question")
>
> You again get the question asked in the minibuffer, but typing
> anything at the prompt causes Emacs to complain that whatever you
> typed is "undefined".  Your only fire escape is to close this session
> with some mouse gesture, or kill it.
>
> What happens here is that, by the time the new thread runs, the main
> (a.k.a. "primary") thread is already idle, i.e. it already called
> read_char, which called wait_reading_process_output, which called
> thread_select.  (In fact, it's _because_ the primary thread called
> thread_select that the new thread was allowed to run at all, since it
> has to acquire the global lock for that, and that is only possible
> when the running thread releases the lock inside thread_select.)  Now,
> when wait_reading_process_output calls thread_select, it prepares the
> mask for file descriptors it will wait for to become ready for
> reading, in this case there's only one descriptor: the keyboard
> descriptor.  So the primary thread is waiting for input from the
> keyboard.
>
> Now the new thread starts running and eventually calls y-or-n-p, which
> calls read_char, which calls wait_reading_process_output.  But when
> this call prepares the mask for the thread_select call, it skips the
> keyboard descriptor, because we only allow a single thread to wait on
> each descriptor, and the keyboard descriptor is already watched by the
> primary thread.  So the new thread ends up not waiting on the keyboard
> input descriptor.
>
> The thread_select call then switches back to the primary thread, and
> the primary thread receives the Y or N character you type.  But it
> doesn't know what to do with it, so it becomes confused.
>
> IOW, user interaction from non-primary threads is currently inherently
> unsafe.

IOW, user interaction from non-primary threads is currently inherently
impossible.

I've tried to understand the mechanism in process.c, and I summarize it
(mainly for my understanding, but also to give you a chance to correct me).

The fd mask is controlled by fd_callback_info, an array (indexed by
fd's) over the struct fd_callback_data. There are two relevant struct
fields: thread and waiting_thread. thread is set in Fset_process_thread
for the infd and outfd file descriptors of the process, it shall be NULL
in the example given above.

waiting_thread is always set to current_thread, in the
compute_input_wait_mask, compute_non_process_wait_mask,
compute_non_keyboard_wait_mask and compute_write_mask functions. The
condition is always, that waiting_thread is either NULL or
current_thread.

A file descriptor, be it for the keyboard or process related or
whatever, is only taken into account during setting an fd_mask, when
either both thread and waiting_thread are NULL, or when one of the
struct fields is set to a thread, being equal to the current
thread. IIUC, thead has precedence over waiting_thread, but this might
be an implementation detail only. So far, it is expected that at least
one of the fields is NULL.

In order to fix the problem of reading input in a non-primary thread, we
need a new function which request I/O control to the current thread. It
needs

* to request control for the keyboard. This could be indicated by a
  signal to the main thread, which is *not* an error signal but a
  special one, let's call it `thread-set-keyboard'.

* to handle this signal in `thread-handle-event', by calling a
  respective function (let's call it `thread--set-keyboard'). This
  function sets the struct event thread to the thread which has
  delivered by the `thread-set-keyboard' signal. Next time fd_mask is
  prepared, the keyboard fd would be taken into account for the
  requesting thread.

* to stop/recall the recent pselect for the main thread in order to free
  the keyboard fd. Don't know how to do this.

* to provide a mechanism which resets the thread struct field of the
  keyboard fd to NULL, in order to let the main thread use the
  keyboard. Here I have also no idea how to do this.

> And then, of course, there's the use case where two threads ask the
> user something via the minibuffer.

This scenario shall also be possible then.

> Thoughts?

Here I am :-)

Best regards, Michael.

bug#25214: 26.0.50; Interacting with user from threads other than the primary

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: 25214@debbugs.gnu.org,  Tom Tromey <tom@tromey.com>
> Date: Mon, 17 Sep 2018 16:06:07 +0200
> 
> The fd mask is controlled by fd_callback_info, an array (indexed by
> fd's) over the struct fd_callback_data. There are two relevant struct
> fields: thread and waiting_thread. thread is set in Fset_process_thread
> for the infd and outfd file descriptors of the process, it shall be NULL
> in the example given above.
> 
> waiting_thread is always set to current_thread, in the
> compute_input_wait_mask, compute_non_process_wait_mask,
> compute_non_keyboard_wait_mask and compute_write_mask functions. The
> condition is always, that waiting_thread is either NULL or
> current_thread.
> 
> A file descriptor, be it for the keyboard or process related or
> whatever, is only taken into account during setting an fd_mask, when
> either both thread and waiting_thread are NULL, or when one of the
> struct fields is set to a thread, being equal to the current
> thread. IIUC, thead has precedence over waiting_thread, but this might
> be an implementation detail only. So far, it is expected that at least
> one of the fields is NULL.

That is correct, AFAIR.

> In order to fix the problem of reading input in a non-primary thread, we
> need a new function which request I/O control to the current thread. It
> needs
> 
> * to request control for the keyboard. This could be indicated by a
>   signal to the main thread, which is *not* an error signal but a
>   special one, let's call it `thread-set-keyboard'.
> 
> * to handle this signal in `thread-handle-event', by calling a
>   respective function (let's call it `thread--set-keyboard'). This
>   function sets the struct event thread to the thread which has
>   delivered by the `thread-set-keyboard' signal. Next time fd_mask is
>   prepared, the keyboard fd would be taken into account for the
>   requesting thread.
> 
> * to stop/recall the recent pselect for the main thread in order to free
>   the keyboard fd. Don't know how to do this.
> 
> * to provide a mechanism which resets the thread struct field of the
>   keyboard fd to NULL, in order to let the main thread use the
>   keyboard. Here I have also no idea how to do this.

It's not all we need (remember all that talk about serializing access
to the echo area?), but it's a beginning.  And the crucial piece is
bullet #3, because thread-signal will not interrupt the pselect call.
And if we are not too lucky, the pselect call could have been made
with a long timeout.  So this is the first obstacle to negotiate.

So what means do we have to interrupt a pselect call from another
thread?

(I'm not sure I understand why you thought we'd need bullet #2, btw.
Once we interrupt the pselect call, the main thread should become
stuck trying to take the global lock, so I don't think we need any
trickery here, because the other thread, which is running, will have
arranged for it to be the current thread.)

Thanks.

bug#25214: 26.0.50; Interacting with user from threads other than the primary

[Eli Zaretskii]

> Date: Mon, 17 Sep 2018 21:41:47 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 25214@debbugs.gnu.org, tom@tromey.com
> 
> So what means do we have to interrupt a pselect call from another
> thread?

One simple idea is not to interrupt a pselect call, but instead to
make sure the main thread never waits for too long for pselect to
return.  We could arrange for acquire_global_lock to maintain a count
of the number of threads that are waiting to take the lock, and if
that number is positive, reduce the timeout we pass to pselect such
that it never waits for more than, say, 1 sec.  Then we need a way for
a non-main thread to wait until the main thread returns from pselect,
at which time the non-main thread could proceed with its own attempt
to read input, while the main thread is stuck waiting for the global
lock.

bug#25214: 26.0.50; Interacting with user from threads other than the primary

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> So what means do we have to interrupt a pselect call from another
>> thread?
>
> One simple idea is not to interrupt a pselect call, but instead to
> make sure the main thread never waits for too long for pselect to
> return.  We could arrange for acquire_global_lock to maintain a count
> of the number of threads that are waiting to take the lock, and if
> that number is positive, reduce the timeout we pass to pselect such
> that it never waits for more than, say, 1 sec.  Then we need a way for
> a non-main thread to wait until the main thread returns from pselect,
> at which time the non-main thread could proceed with its own attempt
> to read input, while the main thread is stuck waiting for the global
> lock.

I'll see whether I could implement something along this idea. But I'm
still learning Emacs' keyboard (more general: FD) handling, so it will
take time.

For waiting of the non-main thread: maybe a mutex and/or condition
variable could do the job.

Best regards, Michael.

bug#25214: 26.0.50; Interacting with user from threads other than the primary

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: 25214@debbugs.gnu.org,  tom@tromey.com
> Date: Tue, 18 Sep 2018 10:29:25 +0200
> 
> > One simple idea is not to interrupt a pselect call, but instead to
> > make sure the main thread never waits for too long for pselect to
> > return.  We could arrange for acquire_global_lock to maintain a count
> > of the number of threads that are waiting to take the lock, and if
> > that number is positive, reduce the timeout we pass to pselect such
> > that it never waits for more than, say, 1 sec.  Then we need a way for
> > a non-main thread to wait until the main thread returns from pselect,
> > at which time the non-main thread could proceed with its own attempt
> > to read input, while the main thread is stuck waiting for the global
> > lock.
> 
> I'll see whether I could implement something along this idea. But I'm
> still learning Emacs' keyboard (more general: FD) handling, so it will
> take time.

It might help to search for timeout_reduced_for_timers in process.c:
that's how we implement a similar logic when timers are waiting to
run.

bug#25214: 26.0.50; Interacting with user from threads other than the primary

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> > One simple idea is not to interrupt a pselect call, but instead to
>> > make sure the main thread never waits for too long for pselect to
>> > return.  We could arrange for acquire_global_lock to maintain a count
>> > of the number of threads that are waiting to take the lock, and if
>> > that number is positive, reduce the timeout we pass to pselect such
>> > that it never waits for more than, say, 1 sec.  Then we need a way for
>> > a non-main thread to wait until the main thread returns from pselect,
>> > at which time the non-main thread could proceed with its own attempt
>> > to read input, while the main thread is stuck waiting for the global
>> > lock.
>> 
>> I'll see whether I could implement something along this idea. But I'm
>> still learning Emacs' keyboard (more general: FD) handling, so it will
>> take time.
>
> It might help to search for timeout_reduced_for_timers in process.c:
> that's how we implement a similar logic when timers are waiting to
> run.

I've played for a while with the keyboard related code, and I must
confess I'm too dumb for this. Sorry, but it is too much, and I have the
feeling I cannot contribute something useful in this area. So I let this
to be fixed by somebody else. This is a pity, because it blocks further
progress in merging the branch feature/tramp-thread-safe into master.

For the time being I concentrate on tasks I feel better suited, like
adding thread support for further file operations.

Best regards, Michael.

bug#25214: 26.0.50; Interacting with user from threads other than the primary

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: 25214@debbugs.gnu.org,  tom@tromey.com
> Date: Thu, 27 Sep 2018 18:15:08 +0200
> 
> >> > One simple idea is not to interrupt a pselect call, but instead to
> >> > make sure the main thread never waits for too long for pselect to
> >> > return.  We could arrange for acquire_global_lock to maintain a count
> >> > of the number of threads that are waiting to take the lock, and if
> >> > that number is positive, reduce the timeout we pass to pselect such
> >> > that it never waits for more than, say, 1 sec.  Then we need a way for
> >> > a non-main thread to wait until the main thread returns from pselect,
> >> > at which time the non-main thread could proceed with its own attempt
> >> > to read input, while the main thread is stuck waiting for the global
> >> > lock.
> >> 
> >> I'll see whether I could implement something along this idea. But I'm
> >> still learning Emacs' keyboard (more general: FD) handling, so it will
> >> take time.
> >
> > It might help to search for timeout_reduced_for_timers in process.c:
> > that's how we implement a similar logic when timers are waiting to
> > run.
> 
> I've played for a while with the keyboard related code, and I must
> confess I'm too dumb for this. Sorry, but it is too much, and I have the
> feeling I cannot contribute something useful in this area. So I let this
> to be fixed by somebody else. This is a pity, because it blocks further
> progress in merging the branch feature/tramp-thread-safe into master.

Can you describe what exactly would you like someone else to
implement?  IOW, what minimal change will allow you to continue your
work in this direction?

Thanks.

bug#25214: 26.0.50; Interacting with user from threads other than the primary

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

> Can you describe what exactly would you like someone else to
> implement?  IOW, what minimal change will allow you to continue your
> work in this direction?

I don't know whether I could do this exactly. But the problem in this
bug report is, that we cannot dedicate keyboard input to a non-main
thread. Usually this is not a problem, I believe that plain editing
shall always happen in the main thread. But sometimes we have the need
to communicate with the user from a non-main thread.

This is a scenario like echoing the prompt "Do you want to delete file
foo?", and the user will type the answer. All this must happen in the
current thread, and also related buffers must be kept (often there is a
help buffer for possible answers).

The point is, that this happens while a pselect is running, which has
dedicated keyboard file descriptors to the main thread already. This
pselect must finish (likely by waiting for the timeout), keyboard
related file descriptors must be rearranged, and a new pselect must be
started which reads the input inside the current thread.

When finished, file descriptors must be rearranged to the main
thread. All of this must happen, while the current thread blocks the
main thread.

Likely, this cannot be done automatically, so we need a function
`set-keyboard-thread' (similar to `set-process-thread'), which is called
for scenarios as described. And since the keyboard must be reset to the
main thread, maybe a macro like this is needed:

(defun with-blocked-keyboard (&rest body)
  `(block-other-threads ;; maybe via a mutex??
     (set-keyboard-thread (current-thread))
     ,@body
     (set-keyboard-thread main-thread)))

It is the fiddling with pselect and the keyboard file descriptors, which
I'm not able to handle.

> Thanks.

Best regards, Michael.

bug#25214: 26.0.50; Interacting with user from threads other than the primary

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> In order to fix the problem of reading input in a non-primary thread, we
>> need a new function which request I/O control to the current thread. It
>> needs
>>
>> * to request control for the keyboard. This could be indicated by a
>>   signal to the main thread, which is *not* an error signal but a
>>   special one, let's call it `thread-set-keyboard'.
>>
>> * to handle this signal in `thread-handle-event', by calling a
>>   respective function (let's call it `thread--set-keyboard'). This
>>   function sets the struct event thread to the thread which has
>>   delivered by the `thread-set-keyboard' signal. Next time fd_mask is
>>   prepared, the keyboard fd would be taken into account for the
>>   requesting thread.
>>
>> * to stop/recall the recent pselect for the main thread in order to free
>>   the keyboard fd. Don't know how to do this.
>>
>> * to provide a mechanism which resets the thread struct field of the
>>   keyboard fd to NULL, in order to let the main thread use the
>>   keyboard. Here I have also no idea how to do this.
>
> It's not all we need (remember all that talk about serializing access
> to the echo area?), but it's a beginning.

Exactly.

> And the crucial piece is bullet #3, because thread-signal will not
> interrupt the pselect call.  And if we are not too lucky, the pselect
> call could have been made with a long timeout.  So this is the first
> obstacle to negotiate.

Yes. There is quit_throw_to_read_char, but this would quit the (main)
thread, which is not what we want.

> So what means do we have to interrupt a pselect call from another
> thread?
>
> (I'm not sure I understand why you thought we'd need bullet #2, btw.
> Once we interrupt the pselect call, the main thread should become
> stuck trying to take the global lock, so I don't think we need any
> trickery here, because the other thread, which is running, will have
> arranged for it to be the current thread.)

Shouldn't there be several read_char calls in parallel? Bullet #2 is
just a machinery to switch to the main thread, that's all. If we could
manipulate the main thread from inside current thread, we wouldn't need this.

> Thanks.

Best regards, Michael.

bug#25214: 26.0.50; Interacting with user from threads other than the primary

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: 25214@debbugs.gnu.org,  tom@tromey.com
> Date: Tue, 18 Sep 2018 10:21:35 +0200
> 
> > (I'm not sure I understand why you thought we'd need bullet #2, btw.
> > Once we interrupt the pselect call, the main thread should become
> > stuck trying to take the global lock, so I don't think we need any
> > trickery here, because the other thread, which is running, will have
> > arranged for it to be the current thread.)
> 
> Shouldn't there be several read_char calls in parallel? Bullet #2 is
> just a machinery to switch to the main thread, that's all. If we could
> manipulate the main thread from inside current thread, we wouldn't need this.

I think we arrived at the conclusion that simultaneous calls to
read_char should be avoided.

bug#25214: #25214 26.0.50; Interacting with user from threads other than the primary

[Zhang Haijun]

Every inputting request from other threads can be treated like an email. Emacs puts all requestes in in-box and show a notification icon(or number) on mode-line. Then user can open the in-box to process an email(inputting request). For example user can run a command like `list-inputting-request` to open a buffer to show all inputting requestes. User can choose any request to process. If an inputting request is chosen by user, then Emacs switches to the thread to process the inputting. 

bug#25214: #25214 26.0.50; Interacting with user from threads other than the primary

[Eli Zaretskii]

> From: Zhang Haijun <ccsmile2008@outlook.com>
> Date: Wed, 6 Feb 2019 06:58:44 +0000
> 
> Every inputting request from other threads can be treated like an email.

The problem is in receiving the input into Emacs in the first place.
It has to be done by some thread, not necessarily the one that is
executing when input arrives.  So we are back to square one.

Thanks.

bug#25214: #25214 26.0.50; Interacting with user from threads other than the primary

[Zhang Haijun]

> 在 2019年2月6日，下午11:41，Eli Zaretskii <eliz@gnu.org> 写道：
> 
>> From: Zhang Haijun <ccsmile2008@outlook.com>
>> Date: Wed, 6 Feb 2019 06:58:44 +0000
>> 
>> Every inputting request from other threads can be treated like an email.
> 
> The problem is in receiving the input into Emacs in the first place.
> It has to be done by some thread, not necessarily the one that is
> executing when input arrives.  So we are back to square one.
> 
> Thanks.

What about the following:

1. Create a fd to receive the input request event. This fd can only be watched by main thread or a special keybord sheduler thread.
2. If a thread need input, it sends an input request event(containning its thread id, prompt …) to the fd and then goto background.
3. The main thread(or keyboard sheduler thread) receives the event and put it into in-box and show a notification to user.
4. User opens the in-box and selects a request. Emacs switches to the thread.

I’m not too familar with emacs’ internal. Maybe I’m wrong.

bug#25214: #25214 26.0.50; Interacting with user from threads other than the primary

[Eli Zaretskii]

> From: Zhang Haijun <ccsmile2008@outlook.com>
> CC: "25214@debbugs.gnu.org" <25214@debbugs.gnu.org>
> Date: Thu, 7 Feb 2019 01:39:54 +0000
> 
> 1. Create a fd to receive the input request event. This fd can only be watched by main thread or a special keybord sheduler thread.
> 2. If a thread need input, it sends an input request event(containning its thread id, prompt …) to the fd and then goto background.

How do you "send input request to the fd"?  A program cannot both read
and write to the same fd within the same process, at least not
portably.

> 3. The main thread(or keyboard sheduler thread) receives the event and put it into in-box and show a notification to user.

If another thread writes to some fd, it means the main thread is not
running, so it cannot receive the event, even if we did find the way
of having an fd which can be read by one thread and written by all the
others.

bug#25214: #25214 26.0.50; Interacting with user from threads other than the primary

[Zhang Haijun]

> 在 2019年2月7日，下午10:29，Eli Zaretskii <eliz@gnu.org> 写道：
> 
> How do you "send input request to the fd"?  A program cannot both read
> and write to the same fd within the same process, at least not
> portably.
> 

Sorry for my poor English and poor expression. I mean a data channel, for example a pair of fd created by pipe() or a pair of sockets created by socketpair(). producer-consumer model.

> If another thread writes to some fd, it means the main thread is not
> running, so it cannot receive the event, even if we did find the way
> of having an fd which can be read by one thread and written by all the
> others.

Another thread send a input request and then be blocked on a lock(for example, keybord lock, one lock per thread). The main thread(or keybord sheduler thread) then can wake up one thread and give keybord to it.

bug#25214: #25214 26.0.50; Interacting with user from threads other than the primary

[Eli Zaretskii]

> From: Zhang Haijun <ccsmile2008@outlook.com>
> CC: "25214@debbugs.gnu.org" <25214@debbugs.gnu.org>
> Date: Thu, 7 Feb 2019 14:56:22 +0000
> 
> 
> Sorry for my poor English and poor expression. I mean a data channel, for example a pair of fd created by pipe() or a pair of sockets created by socketpair(). producer-consumer model.
> 
> > If another thread writes to some fd, it means the main thread is not
> > running, so it cannot receive the event, even if we did find the way
> > of having an fd which can be read by one thread and written by all the
> > others.
> 
> Another thread send a input request and then be blocked on a lock(for example, keybord lock, one lock per thread). The main thread(or keybord sheduler thread) then can wake up one thread and give keybord to it.

Unless the thread that listens to these requests is never blocked, I
don't see how this could work.

And in any case, what should a non-main thread do if it wants to ask a
question?  Asking a question involves displaying the question before
reading the answer.  You only talked about the second part.

bug#25214: #25214 26.0.50; Interacting with user from threads other than the primary

[Zhang Haijun]

> 在 2019年2月8日，上午1:25，Eli Zaretskii <eliz@gnu.org> 写道：
> 
>> From: Zhang Haijun <ccsmile2008@outlook.com>
>> CC: "25214@debbugs.gnu.org" <25214@debbugs.gnu.org>
>> Date: Thu, 7 Feb 2019 14:56:22 +0000
>> 
>> 
>> Sorry for my poor English and poor expression. I mean a data channel, for example a pair of fd created by pipe() or a pair of sockets created by socketpair(). producer-consumer model.
>> 
>>> If another thread writes to some fd, it means the main thread is not
>>> running, so it cannot receive the event, even if we did find the way
>>> of having an fd which can be read by one thread and written by all the
>>> others.
>> 
>> Another thread send a input request and then be blocked on a lock(for example, keybord lock, one lock per thread). The main thread(or keybord sheduler thread) then can wake up one thread and give keybord to it.
> 
> Unless the thread that listens to these requests is never blocked, I
> don't see how this could work.
> 
It listens to these requests, so it will be waked up when there is a new request arriving. It then reads the request and put it into in-box.

When user want to process an input, the following function will be called:

void Process_an_input_request(input_request)
{
	// 0. get the thread of the request
	thread_id = input_request->thread_id;
	the_thread = get_thread(thread_id);

	// 1. wake up the thread, and give keyboard to it
	Release_lock(the_thread->keyboard_using_lock);
	// 2. wait for the thread to finish
	Acquire_lock(the_thread->keyboard_finished_lock);  // will be blocked until the thread finishes the input task

	// 3. reclaim the keyboard
	Acquire_lock(the_thread->keyboard_using_lock);
	Release_lock(the_thread->keyboard_finished_lock);

	// ...
}

The above function will blocks the main thread(or keyboard sheduler thread) until the input is finished. Don’t known is it OK for emacs internal.


> And in any case, what should a non-main thread do if it wants to ask a
> question?  Asking a question involves displaying the question before
> reading the answer.  You only talked about the second part.

It can call a function like the following:

int Ask_a_question_wrapper(prompt)
{
	Send_input_request_event(Get_thread_id(), prompt);

	// 1. request the keyboard
	Acquire_lock(Current_thread()->keyboard_finished_lock);
	Acquire_lock(Current_thread()->keyboard_using_lock); // will be blocked until the main thread wake us up.

	// 2. do the real input
	result = Ask_a_question(prompt);

	// 3. give back the keyboard
	Release_lock(Current_thread()->keyboard_using_lock);
	Release_lock(Current_thread()->keyboard_finished_lock);

	return result;
}

The keyboard_using_lock shoud be acquired by the main thread initially.
Is there a simple way to let a specified thread run and wait for it to finish a task?
Maybe the client-server model is simpler, like TCP’s three way handshake.

bug#25214: #25214 26.0.50; Interacting with user from threads other than the primary

[Zhang Haijun]

> 在 2019年2月7日，上午9:39，张海君 <ccsmile2008@outlook.com> 写道：
> 
> 1. Create a fd to receive the input request event. This fd can only be watched by main thread or a special keybord sheduler thread.

The fd may be a socket created by socketpair().