Return value of finished threads

Return value of finished threads

[Michael Albinus]

Hi,

these days I'm working on making Tramp thread-safe. This works already,
but is only part of the job.

The other part is *using* threads. My first use case I'll try to
implement is `find-file' with wildcards. That is, for every single file
to be loaded, an own thread is created. A user indicates this
interactively by a prefix argument like "C-u M-x find-file ...".

What I'm missing is a simple possibility to collect the return values of
the respective finished threads (`find-file-noselect' calls), which is a
buffer or a list of buffers. The documentation recommends the use of
global variables, which would be inconvenient for many threads to supervise.

So I'm wondering if we could extend thread support by an indication of
the thread result, when finished. One idea would be that `thread-join'
returns with that value.

Another idea, even broader, would be to extend `condition-notify' by an
argument, which passes a Lisp object to another thread. The
corresponding `condition-wait' would return that value.

What do people think?

Best regards, Michael.

Re: Return value of finished threads

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Date: Fri, 20 Jul 2018 16:56:48 +0200
> 
> What I'm missing is a simple possibility to collect the return values of
> the respective finished threads (`find-file-noselect' calls), which is a
> buffer or a list of buffers. The documentation recommends the use of
> global variables, which would be inconvenient for many threads to supervise.
> 
> So I'm wondering if we could extend thread support by an indication of
> the thread result, when finished. One idea would be that `thread-join'
> returns with that value.
> 
> Another idea, even broader, would be to extend `condition-notify' by an
> argument, which passes a Lisp object to another thread. The
> corresponding `condition-wait' would return that value.

Do you have an implementation in mind?  Because in general, passing
variables between threads in C also needs global variables...

Re: Return value of finished threads

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

> Do you have an implementation in mind?  Because in general, passing
> variables between threads in C also needs global variables...

Not (yet).

I've written this from my use case point of view. It is obvious that
global variables are needed (maybe kept in a hash or so), but I rather
give this task to threads.c, instead writing it myself (and urging
anybody else to write it herself).

Best regards, Michael.

Re: Return value of finished threads

[Clément Pit-Claudel]

On 2018-07-20 11:15, Michael Albinus wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
> 
>> Do you have an implementation in mind?  Because in general, passing
>> variables between threads in C also needs global variables...
> 
> Not (yet).
> 
> I've written this from my use case point of view. It is obvious that
> global variables are needed (maybe kept in a hash or so), but I rather
> give this task to threads.c, instead writing it myself (and urging
> anybody else to write it herself).

On the ELisp side, I wonder if it'd make sense to attach a plist to each thread.  With this you would do (thread-put t 'result (returned value)) from the thread, and on the waiting side you'd run thread-join before accessing the result with (thread-get t 'result)

Clément.

Re: Return value of finished threads

[Noam Postavsky]

On 20 July 2018 at 10:56, Michael Albinus <michael.albinus@gmx.de> wrote:

> What I'm missing is a simple possibility to collect the return values of
> the respective finished threads (`find-file-noselect' calls), which is a
> buffer or a list of buffers. The documentation recommends the use of
> global variables, which would be inconvenient for many threads to supervise.

> What do people think?

I think you are looking for something along the lines of
promises/futures. Maybe thunk.el can be used for this (note, if trying
*scratch* remember to set lexical-binding):

;; -*- lexical-binding: t -*-
; (setq lexical-binding t) ; for *scratch*

(require 'thunk)
(defmacro thunk-delay-in-thread (&rest body)
  (declare (debug body))
  (let ((result-var (make-symbol "result")))
    `(let* ((,result-var nil)
            (thread (make-thread
                     (lambda ()
                       (setq ,result-var ,(macroexp-progn body))))))
       (thunk-delay (progn (thread-join thread) ,result-var)))))

(let ((thunks
       (mapcar
        (lambda (x) (thunk-delay-in-thread
                     (message "computing square of %d" x) (* x x)))
        (number-sequence 1 10))))
  ;; Run this thread first, just to make things more interesting.
  (thunk-force (nth 5 thunks))
  ;; Get all the values.
  (mapcar #'thunk-force thunks))

Re: Return value of finished threads

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Fri, 20 Jul 2018 12:17:04 -0400
> 
> On the ELisp side, I wonder if it'd make sense to attach a plist to each thread.  With this you would do (thread-put t 'result (returned value)) from the thread, and on the waiting side you'd run thread-join before accessing the result with (thread-get t 'result)

When the thread function returns, the thread is dead, and I'm not sure
we should encourage using plists of dead threads.

Re: Return value of finished threads

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@gmail.com>
> Date: Fri, 20 Jul 2018 12:17:46 -0400
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> (require 'thunk)
> (defmacro thunk-delay-in-thread (&rest body)
>   (declare (debug body))
>   (let ((result-var (make-symbol "result")))
>     `(let* ((,result-var nil)
>             (thread (make-thread
>                      (lambda ()
>                        (setq ,result-var ,(macroexp-progn body))))))
>        (thunk-delay (progn (thread-join thread) ,result-var)))))
> 
> (let ((thunks
>        (mapcar
>         (lambda (x) (thunk-delay-in-thread
>                      (message "computing square of %d" x) (* x x)))
>         (number-sequence 1 10))))
>   ;; Run this thread first, just to make things more interesting.
>   (thunk-force (nth 5 thunks))
>   ;; Get all the values.
>   (mapcar #'thunk-force thunks))

This sounds gross to me, FWIW.

Re: Return value of finished threads

[Tom Tromey]

>>>>> "Michael" == Michael Albinus <michael.albinus@gmx.de> writes:

Michael> What I'm missing is a simple possibility to collect the return values of
Michael> the respective finished threads (`find-file-noselect' calls), which is a
Michael> buffer or a list of buffers.

Michael> So I'm wondering if we could extend thread support by an indication of
Michael> the thread result, when finished. One idea would be that `thread-join'
Michael> returns with that value.

This doesn't seem all that hard to implement.
Also if the thread exited due to an exception of some kind, this could
be propagated by thread-join as well.

Michael> The documentation recommends the use of
Michael> global variables, which would be inconvenient for many threads to supervise.

Or you can use a queue of some kind and have each worker push its result
there.

Tom

Re: Return value of finished threads

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> (require 'thunk)
>> (defmacro thunk-delay-in-thread (&rest body)
>>   (declare (debug body))
>>   (let ((result-var (make-symbol "result")))
>>     `(let* ((,result-var nil)
>>             (thread (make-thread
>>                      (lambda ()
>>                        (setq ,result-var ,(macroexp-progn body))))))
>>        (thunk-delay (progn (thread-join thread) ,result-var)))))
>> 
>> (let ((thunks
>>        (mapcar
>>         (lambda (x) (thunk-delay-in-thread
>>                      (message "computing square of %d" x) (* x x)))
>>         (number-sequence 1 10))))
>>   ;; Run this thread first, just to make things more interesting.
>>   (thunk-force (nth 5 thunks))
>>   ;; Get all the values.
>>   (mapcar #'thunk-force thunks))
>
> This sounds gross to me, FWIW.

Yes. See my other mail for an implementation in threads.c.

Best regards, Michael.

Re: Return value of finished threads

[Michael Albinus]

[-- Attachment #1: Type: text/plain, Size: 1300 bytes --]

Tom Tromey <tom@tromey.com> writes:

Hi Tom,

> Michael> So I'm wondering if we could extend thread support by an
> Michael> indication of the thread result, when finished. One idea
> Michael> would be that `thread-join' returns with that value.
>
> This doesn't seem all that hard to implement.

Yep. The appended patch does it for me. Maybe you could check whether
I'm missing something?

> Also if the thread exited due to an exception of some kind, this could
> be propagated by thread-join as well.

Yeah, this would be a bonus. I've failed to implement it, so I let it to
somebody else. I don't need it first-hand.

> Michael> The documentation recommends the use of global variables,
> Michael> which would be inconvenient for many threads to supervise.
>
> Or you can use a queue of some kind and have each worker push its
> result there.

Yes, but you must indicate the worker threads where to push the
result. Imagine find-file would support threads for wildcard file names,
and somebody calls find-file this way twice, both running concurrently.
Every worker thread must know, whether to push the result into the
global variable associated with the first call or the second one. Still
overhead to implement; I'll prefer that thread-join returns the result.

> Tom

Best regards, Michael.


[-- Attachment #2: Type: text/plain, Size: 3286 bytes --]

diff --git a/src/thread.c b/src/thread.c
index 754d286e9f..e2d16df7cb 100644
--- a/src/thread.c
+++ b/src/thread.c
@@ -681,7 +681,7 @@ invoke_thread_function (void)
 {
   ptrdiff_t count = SPECPDL_INDEX ();
 
-  Ffuncall (1, &current_thread->function);
+  current_thread->result = Ffuncall (1, &current_thread->function);
   return unbind_to (count, Qnil);
 }
 
@@ -789,6 +789,7 @@ If NAME is given, it must be a string; it names the new thread.  */)
   new_thread->m_last_thing_searched = Qnil; /* copy from parent? */
   new_thread->m_saved_last_thing_searched = Qnil;
   new_thread->m_current_buffer = current_thread->m_current_buffer;
+  new_thread->result = Qnil;
   new_thread->error_symbol = Qnil;
   new_thread->error_data = Qnil;
   new_thread->event_object = Qnil;
@@ -933,9 +934,9 @@ thread_join_callback (void *arg)
 
 DEFUN ("thread-join", Fthread_join, Sthread_join, 1, 1, 0,
        doc: /* Wait for THREAD to exit.
-This blocks the current thread until THREAD exits or until
-the current thread is signaled.
-It is an error for a thread to try to join itself.  */)
+This blocks the current thread until THREAD exits or until the current
+thread is signaled.  It returns the result of the THREAD function.  It
+is an error for a thread to try to join itself.  */)
   (Lisp_Object thread)
 {
   struct thread_state *tstate;
@@ -949,7 +950,7 @@ It is an error for a thread to try to join itself.  */)
   if (thread_alive_p (tstate))
     flush_stack_call_func (thread_join_callback, tstate);
 
-  return Qnil;
+  return tstate->result;
 }
 
 DEFUN ("all-threads", Fall_threads, Sall_threads, 0, 0, 0,
@@ -1017,6 +1018,7 @@ init_main_thread (void)
   main_thread.m_saved_last_thing_searched = Qnil;
   main_thread.name = Qnil;
   main_thread.function = Qnil;
+  main_thread.result = Qnil;
   main_thread.error_symbol = Qnil;
   main_thread.error_data = Qnil;
   main_thread.event_object = Qnil;
@@ -1090,8 +1092,7 @@ syms_of_threads (void)
   DEFSYM (Qmutexp, "mutexp");
   DEFSYM (Qcondition_variable_p, "condition-variable-p");
 
-  DEFVAR_LISP ("main-thread",
-	       Vmain_thread,
+  DEFVAR_LISP ("main-thread", Vmain_thread,
     doc: /* The main thread of Emacs.  */);
 #ifdef THREADS_ENABLED
   XSETTHREAD (Vmain_thread, &main_thread);
diff --git a/src/thread.h b/src/thread.h
index c10e5ecb75..922eea6217 100644
--- a/src/thread.h
+++ b/src/thread.h
@@ -52,6 +52,9 @@ struct thread_state
   /* The thread's function.  */
   Lisp_Object function;
 
+  /* The thread's result, if function has finished.  */
+  Lisp_Object result;
+
   /* If non-nil, this thread has been signaled.  */
   Lisp_Object error_symbol;
   Lisp_Object error_data;
diff --git a/test/src/thread-tests.el b/test/src/thread-tests.el
index a447fb3914..ddf9e987cc 100644
--- a/test/src/thread-tests.el
+++ b/test/src/thread-tests.el
@@ -100,9 +100,9 @@ threads-test-thread1
    (progn
      (setq threads-test-global nil)
      (let ((thread (make-thread #'threads-test-thread1)))
-       (thread-join thread)
-       (and threads-test-global
-	    (not (thread-alive-p thread)))))))
+       (and (= (thread-join thread) 23)
+            (= threads-test-global 23)
+            (not (thread-alive-p thread)))))))
 
 (ert-deftest threads-join-self ()
   "Cannot `thread-join' the current thread."

Re: Return value of finished threads

[Tom Tromey]

>>>>> "Michael" == Michael Albinus <michael.albinus@gmx.de> writes:

Tom> This doesn't seem all that hard to implement.

Michael> Yep. The appended patch does it for me. Maybe you could check whether
Michael> I'm missing something?

I think the way threads are marked has to be changed.
I believe that, currently, once a thread exits, its fields are no longer
marked.  However, with this change we want to be sure that the result is
marked for a dead thread.

Tom

Re: Return value of finished threads

[Michael Albinus]

Tom Tromey <tom@tromey.com> writes:

> I think the way threads are marked has to be changed.
> I believe that, currently, once a thread exits, its fields are no longer
> marked.  However, with this change we want to be sure that the result is
> marked for a dead thread.

Yes, likely until a thread-join has happened. And a second thread-join
on the same thread shall always return ni.

How long do we keep the results of dead threads, for which no
thread-join happens? Do we need a Lisp cleanup function, which purges
all dead threads results? Or does this the GC, when no Lisp object holds
the dead thread any longer?

(Btw, I believe it will be much better when you implement this. It is
still hard for me to read and modify threads.c)

> Tom

Best regards, Michael.

Re: Return value of finished threads

[Tom Tromey]

>>>>> "Michael" == Michael Albinus <michael.albinus@gmx.de> writes:

Michael> Yes, likely until a thread-join has happened. And a second thread-join
Michael> on the same thread shall always return ni.

I was wondering which semantics would be preferred here.
Having each thread-join return the same result doesn't seem so bad either.

Michael> How long do we keep the results of dead threads, for which no
Michael> thread-join happens? Do we need a Lisp cleanup function, which purges
Michael> all dead threads results? Or does this the GC, when no Lisp object holds
Michael> the dead thread any longer?

I didn't look deeply but I think instead of mark_threads, alloc.c would
need to mark any thread it runs across; and then mark_one_thread could
handle exited threads specially.

Michael> (Btw, I believe it will be much better when you implement this. It is
Michael> still hard for me to read and modify threads.c)

I will see if I can find some time.  But I think your patch is close so
you should keep going :-)

Tom

Re: Return value of finished threads

[Michael Albinus]

Tom Tromey <tom@tromey.com> writes:

> Michael> Yes, likely until a thread-join has happened. And a second
> Michael> thread-join on the same thread shall always return ni.
>
> I was wondering which semantics would be preferred here.
> Having each thread-join return the same result doesn't seem so bad either.

I have no strong preference. But returning the result only once has the
advantage, that you could remove all thread data from a dead thread,
once the result has been shown.

> Michael> How long do we keep the results of dead threads, for which no
> Michael> thread-join happens? Do we need a Lisp cleanup function, which purges
> Michael> all dead threads results? Or does this the GC, when no Lisp object holds
> Michael> the dead thread any longer?
>
> I didn't look deeply but I think instead of mark_threads, alloc.c would
> need to mark any thread it runs across; and then mark_one_thread could
> handle exited threads specially.

This goes into the darkness I'm not familiar with.

> Michael> (Btw, I believe it will be much better when you implement this. It is
> Michael> still hard for me to read and modify threads.c)
>
> I will see if I can find some time.  But I think your patch is close so
> you should keep going :-)

Sure. I use my patch locally to work on Tramp threads, but I won't push
it upstream.

> Tom

Best regards, Michael.

Re: Return value of finished threads

[Tom Tromey]

>> I was wondering which semantics would be preferred here.
>> Having each thread-join return the same result doesn't seem so bad either.

Michael> I have no strong preference. But returning the result only once has the
Michael> advantage, that you could remove all thread data from a dead thread,
Michael> once the result has been shown.

I think it's somewhat better to keep the value.  This avoids races or
the need for an extra rendezvous of some kind if there are multiple
joiners.

Keeping the value does mean that it is not GC'd.  However, normally the
idea would be to accomplish this by forgetting the dead thread in the
first place; which I would imagine is the most normal course of action.
So I tend to think this isn't a major concern.

Michael> How long do we keep the results of dead threads, for which no
Michael> thread-join happens? Do we need a Lisp cleanup function, which purges
Michael> all dead threads results? Or does this the GC, when no Lisp object holds
Michael> the dead thread any longer?

Tom> I didn't look deeply but I think instead of mark_threads, alloc.c would
Tom> need to mark any thread it runs across; and then mark_one_thread could
Tom> handle exited threads specially.

Michael> This goes into the darkness I'm not familiar with.

Good news -- it turns out I was mistaken about this.

The ordinary Lisp_Object fields of a thread are marked by mark_object,
because it is a pseudovector.  The other fields are what are marked
specially by mark_threads -- and these only need to be marked for live
threads.

I think a documentation update and a NEWS entry would be required.

thanks,
Tom

Re: Return value of finished threads

[Michael Albinus]

Tom Tromey <tom@tromey.com> writes:

Hi Tom,

> The ordinary Lisp_Object fields of a thread are marked by mark_object,
> because it is a pseudovector.  The other fields are what are marked
> specially by mark_threads -- and these only need to be marked for live
> threads.

I'm still lost. Will you provide a respective patch, or does it mean I
could push my patch as-it-is?

> I think a documentation update and a NEWS entry would be required.

Will do, once the change has arrived master. I need to document the
changes for Tramp thread-safe and for find-file* anyway.

> thanks,
> Tom

Best regards, Michael.

Re: Return value of finished threads

[Tom Tromey]

>>>>> "Michael" == Michael Albinus <michael.albinus@gmx.de> writes:

Michael> I'm still lost. Will you provide a respective patch, or does it mean I
Michael> could push my patch as-it-is?

I think it's already fine.

Tom

Re: Return value of finished threads

[Brett Gilio]

Michael Albinus writes:
> How long do we keep the results of dead threads, for which no
> thread-join happens? Do we need a Lisp cleanup function, which 
> purges
> all dead threads results? Or does this the GC, when no Lisp 
> object holds
> the dead thread any longer?

I was wondering of this, myself. Is the garbage collector of emacs 
lisp
similar to the Boehm collector style of Guile? Does anybody have 
any
information to pass along on the emacs lisp GC?

-- 
Brett M. Gilio
Free Software Foundation, Member
https://parabola.nu | https://emacs.org

Re: Return value of finished threads

[Tom Tromey]

>>>>> "Brett" == Brett Gilio <brettg@posteo.net> writes:

Brett> I was wondering of this, myself. Is the garbage collector of
Brett> emacs lisp similar to the Boehm collector style of Guile? Does
Brett> anybody have any information to pass along on the emacs lisp GC?

The Emacs GC is a fairly ordinary mark/sweep GC.  I'd say it is nearly
the most straightforward garbage collector you could imagine; or at
least started that way before features like weak hash maps and
finalizers were added :)

It is vaguely similar to the Boehm GC.  The Boehm GC is best known for
doing conservative marking (but it has several modes and isn't limited
to this); whereas the Emacs GC conservatively marks the stack but not
the heap.

If you want to learn more about the Emacs GC in particular, alloc.c is
not that hard to read.  Start with the function garbage_collect_1.

If you need some background in garbage collection in general, the Jones
& Lins book "Garbage Collection" is very good.

Tom

Re: Return value of finished threads

[Michael Albinus]

Tom Tromey <tom@tromey.com> writes:

Hi Tom,

> Michael> I'm still lost. Will you provide a respective patch, or does
> Michael> it mean I could push my patch as-it-is?
>
> I think it's already fine.

I've pushed this to master, including documentation and NEWS.

I've also added some code for signal propagation of the finished thread
to the current one. This works OK for the quit signal (see changed
threads-mutex-signal test), but this does not work for ordinary errors
(see new test threads-join-error, I've marked it as :unstable for the
time being). I guess that the handlers catch the error somehow; maybe
you could have a look on this.

For this reason, I haven't documented signal propagation yet.

I'm also uncertain whether it is desired to propagate signals via
thread-join in all use cases. Maybe we add an optional NOSIGNAL argument
to this function?

> Tom

Best regards, Michael.

Re: Return value of finished threads

[Brett Gilio]

Tom Tromey writes:

>>>>>> "Brett" == Brett Gilio <brettg@posteo.net> writes:
>
> Brett> I was wondering of this, myself. Is the garbage collector 
> of
> Brett> emacs lisp similar to the Boehm collector style of Guile? 
> Does
> Brett> anybody have any information to pass along on the emacs 
> lisp GC?
>
> The Emacs GC is a fairly ordinary mark/sweep GC.  I'd say it is 
> nearly
> the most straightforward garbage collector you could imagine; or 
> at
> least started that way before features like weak hash maps and
> finalizers were added :)
>
> It is vaguely similar to the Boehm GC.  The Boehm GC is best 
> known for
> doing conservative marking (but it has several modes and isn't 
> limited
> to this); whereas the Emacs GC conservatively marks the stack 
> but not
> the heap.
>
> If you want to learn more about the Emacs GC in particular, 
> alloc.c is
> not that hard to read.  Start with the function 
> garbage_collect_1.
>
> If you need some background in garbage collection in general, 
> the Jones
> & Lins book "Garbage Collection" is very good.
>
> Tom


Thank you Tom, I read through the GC and it is mostly pretty 
straight
forward, with some nice quirks.


-- 
Brett M. Gilio
Free Software Foundation, Member
https://parabola.nu | https://emacs.org