make-thread with lambda form instead of function symbol

make-thread with lambda form instead of function symbol

[Eric Abrahamsen]

Okay, one more thread question...

I'm trying to do something that seems like it would be a normal use
case: spawn a series of threads which call the same function using
different external processes. Practically what this means is that I want
to pass a function-plus-argument form to make-thread, not a function
symbol. Something like:

(let* ((results)
       (sources '(source1 source2))
       (threads
	(mapcar
	 (lambda (s)
	   (make-thread
	    (funcall
	     (lambda ()
	       (push (get-stuff-from-source s) results)))))
	 sources)))
  (mapc #'thread-join threads)
  results)

The (funcall (lambda () thing was the only way I could get anything but
nil out of the thread functions. I think I'm fooling myself, though: so
far as I can tell, `get-stuff-from-source' is fully evaluated before the
thread is made, and nothing at all happens during the #'thread-join
loop.

Is it possible to give make-thread anything but a function symbol? The
only other thing I could think of was looping over the sources and
making ad-hoc symbols:

(fset (make-symbol (format "%s-dummy-function" (source-name source)))
      (lambda () (push (get-stuff-from-source source) results)))

That seems ugly, but perhaps not that bad.

Thanks in advance,
Eric

Re: make-thread with lambda form instead of function symbol

[Noam Postavsky]

On Sun, Apr 16, 2017 at 12:05 PM, Eric Abrahamsen
<eric@ericabrahamsen.net> wrote:
> I'm trying to do something that seems like it would be a normal use
> case: spawn a series of threads which call the same function using
> different external processes. Practically what this means is that I want
> to pass a function-plus-argument form to make-thread, not a function
> symbol. Something like:
>
> (let* ((results)
>        (sources '(source1 source2))
>        (threads
>         (mapcar
>          (lambda (s)
>            (make-thread
>             (funcall
>              (lambda ()
>                (push (get-stuff-from-source s) results)))))
>          sources)))
>   (mapc #'thread-join threads)
>   results)
>
> The (funcall (lambda () thing was the only way I could get anything but
> nil out of the thread functions. I think I'm fooling myself, though: so
> far as I can tell, `get-stuff-from-source' is fully evaluated before the
> thread is made, and nothing at all happens during the #'thread-join
> loop.

Just drop the funcall: (make-thread (lambda () ...)) should do what
you want, assuming you have lexical-binding set (if not, you can
construct a lambda-list with backquote or similar).

Re: make-thread with lambda form instead of function symbol

[Eric Abrahamsen]

Noam Postavsky <npostavs@users.sourceforge.net> writes:

> On Sun, Apr 16, 2017 at 12:05 PM, Eric Abrahamsen
> <eric@ericabrahamsen.net> wrote:
>> I'm trying to do something that seems like it would be a normal use
>> case: spawn a series of threads which call the same function using
>> different external processes. Practically what this means is that I want
>> to pass a function-plus-argument form to make-thread, not a function
>> symbol. Something like:
>>
>> (let* ((results)
>>        (sources '(source1 source2))
>>        (threads
>>         (mapcar
>>          (lambda (s)
>>            (make-thread
>>             (funcall
>>              (lambda ()
>>                (push (get-stuff-from-source s) results)))))
>>          sources)))
>>   (mapc #'thread-join threads)
>>   results)
>>
>> The (funcall (lambda () thing was the only way I could get anything but
>> nil out of the thread functions. I think I'm fooling myself, though: so
>> far as I can tell, `get-stuff-from-source' is fully evaluated before the
>> thread is made, and nothing at all happens during the #'thread-join
>> loop.
>
> Just drop the funcall: (make-thread (lambda () ...)) should do what
> you want, assuming you have lexical-binding set (if not, you can
> construct a lambda-list with backquote or similar).

Thanks for the response. The first thing I learned here was that
lexical-binding was nil in *scratch*, but t in the source file I'm
writing -- not a great testing setup.

I'm still not seeing the results I expect, though. Here's a dumb test:

(setq lexical-binding t)

(let ((threads
       (mapcar
	(lambda (el)
	  (make-thread
	   (lambda ()
	     (push (cl-incf el) results))))
	'(1 2 3)))
      results)
  (mapc #'thread-join threads)
  results)

This gives me nil.

(Incidentally, if I put this in a function and edebug it, it tells me
edebug will stop at the next break point, and then enters a level of
recursive editing I can't escape from: C-M-c gives me "No catch for tag:
exit, nil".)

Should the above example work?

Thanks again,
Eric

Re: make-thread with lambda form instead of function symbol

[Noam Postavsky]

On Sun, Apr 16, 2017 at 2:11 PM, Eric Abrahamsen
<eric@ericabrahamsen.net> wrote:
>
> (setq lexical-binding t)
>
> (let ((threads
>        (mapcar
>         (lambda (el)
>           (make-thread
>            (lambda ()
>              (push (cl-incf el) results))))
>         '(1 2 3)))
>       results)
>   (mapc #'thread-join threads)
>   results)
>
> This gives me nil.
>
> (Incidentally, if I put this in a function and edebug it, it tells me
> edebug will stop at the next break point, and then enters a level of
> recursive editing I can't escape from: C-M-c gives me "No catch for tag:
> exit, nil".)
>
> Should the above example work?

No, check the compile warnings:

a.el:7:33:Warning: reference to free variable ‘results’
a.el:9:7:Warning: assignment to free variable ‘results’

Not sure about the edebug thing, probably it doesn't handle
cross-thread stepping.

The below returns (2 3 4) or sometimes (3 4 2).

(let* ((results nil)
       (threads
        (mapcar
         (lambda (el)
           (make-thread
            (lambda ()
              (push (cl-incf el) results))))
         '(1 2 3))))
  (mapc #'thread-join threads)
  results)

Re: make-thread with lambda form instead of function symbol

[Eric Abrahamsen]

Noam Postavsky <npostavs@users.sourceforge.net> writes:

> On Sun, Apr 16, 2017 at 2:11 PM, Eric Abrahamsen
> <eric@ericabrahamsen.net> wrote:
>>
>> (setq lexical-binding t)
>>
>> (let ((threads
>>        (mapcar
>>         (lambda (el)
>>           (make-thread
>>            (lambda ()
>>              (push (cl-incf el) results))))
>>         '(1 2 3)))
>>       results)
>>   (mapc #'thread-join threads)
>>   results)
>>
>> This gives me nil.
>>
>> (Incidentally, if I put this in a function and edebug it, it tells me
>> edebug will stop at the next break point, and then enters a level of
>> recursive editing I can't escape from: C-M-c gives me "No catch for tag:
>> exit, nil".)
>>
>> Should the above example work?
>
> No, check the compile warnings:
>
> a.el:7:33:Warning: reference to free variable ‘results’
> a.el:9:7:Warning: assignment to free variable ‘results’

Aha! That makes sense now that I see it. The thread function starts
executing immediately, so of course it needs to be able to see
"results". Thanks for that.

> Not sure about the edebug thing, probably it doesn't handle
> cross-thread stepping.

I can imagine it would be complicated.

Thanks again,
Eric

Re: make-thread with lambda form instead of function symbol

[Eric Abrahamsen]

[-- Attachment #1: Type: text/plain, Size: 1878 bytes --]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Okay, one more thread question...

Well, here's the long story: I'm messing with Gnus's nnir search
routines, trying to make a general search query language that can be
translated into backend-specific queries depending on which groups
you've selected to search. The idea is you could mark both an IMAP group
and an nnmaildir group (indexed by notmuch), enter a single search
query, the query would be transformed into backend-appropriate versions,
and you'd get one search group containing messages returned by backends.

So then I figured: if we're searching different backends at once, and
each backend does its heavy lifting in an external process, then threads
would allow those processes to do their work concurrently, and results
could come back faster.

(If I can get this to work, I hope to look at getting Gnus to fetch new
news concurrently, as well.)

But! My current implementation is causing Emacs to segfault. The actual
function looks like this:

(defun nnir-run-query (specs)
  (let* ((results [])
	 (threads
	  (mapcar
	   (lambda (x)
	     (let* ((server (car x))
		    (search-engine (nnir-server-to-search-engine server)))
	       (make-thread
		(lambda ()
		  (setq results
			(vconcat
			 (nnir-run-search
			  search-engine
			  server
			  (cdr (assq 'nnir-query-spec specs))
			  (cadr x))
			 results))))))
	   (cdr (assq 'nnir-group-spec specs)))))
    (mapc #'thread-join threads)
    results))

I'm testing with a single IMAP server query (ie, only one thread is
being created). The nnir-run-search function for the IMAP backend ends
up calling nnimap-send-command--> nnimap-wait-for-response-->
nnheader-accept-process-output--> accept-process-output, so I'm
expecting that's where the thread yields.

I'm attaching the traceback here. Dunno if it's something I've done
wrong...

Thanks,
Eric


[-- Attachment #2: log.txt --]
[-- Type: text/plain, Size: 6545 bytes --]

#0  0x000000000058879c in terminate_due_to_signal (sig=6, backtrace_limit=2147483647) at emacs.c:363
#1  0x000000000061dbf4 in die (msg=0x74e3fd "STRINGP (a)", file=0x74e330 "lisp.h", line=1276) at alloc.c:7339
#2  0x0000000000582dfd in XSTRING (a=...) at lisp.h:1276
#3  0x0000000000582e44 in SDATA (string=...) at lisp.h:1327
#4  0x000000000069f953 in Faccept_process_output (process=..., seconds=..., millisec=..., just_this_one=...) at process.c:4566
#5  0x00000000006430f7 in funcall_subr (subr=0xd74628 <Saccept_process_output>, numargs=3, args=0x7fffe5987f10) at eval.c:2826
#6  0x0000000000642bda in Ffuncall (nargs=4, args=0x7fffe5987f08) at eval.c:2743
#7  0x0000000000691dd2 in exec_byte_code (bytestr=..., vector=..., maxdepth=..., args_template=..., nargs=0, args=0x0) at bytecode.c:641
#8  0x0000000000643cb9 in funcall_lambda (fun=..., nargs=1, arg_vector=0x577f1f5)
    at eval.c:3021
#9  0x0000000000642c0e in Ffuncall (nargs=2, args=0x7fffe5988638) at eval.c:2745
#10 0x0000000000691dd2 in exec_byte_code (bytestr=..., vector=..., maxdepth=..., args_template=..., nargs=0, args=0x0) at bytecode.c:641
#11 0x0000000000643cb9 in funcall_lambda (fun=..., nargs=1, arg_vector=0x58b7325)
    at eval.c:3021
#12 0x0000000000642c0e in Ffuncall (nargs=2, args=0x7fffe5988e58) at eval.c:2745
#13 0x0000000000691dd2 in exec_byte_code (bytestr=..., vector=..., maxdepth=..., args_template=..., nargs=0, args=0x0) at bytecode.c:641
#14 0x0000000000643cb9 in funcall_lambda (fun=..., nargs=1, arg_vector=0x58b71ed)
    at eval.c:3021
#15 0x0000000000642c0e in Ffuncall (nargs=2, args=0x7fffe5989558) at eval.c:2745
#16 0x0000000000691dd2 in exec_byte_code (bytestr=..., vector=..., maxdepth=..., args_template=..., nargs=0, args=0x0) at bytecode.c:641
#17 0x0000000000643cb9 in funcall_lambda (fun=..., nargs=3, arg_vector=0x58b713d)
    at eval.c:3021
#18 0x0000000000642c0e in Ffuncall (nargs=4, args=0x7fffe5989ca8) at eval.c:2745
#19 0x0000000000691dd2 in exec_byte_code (bytestr=..., vector=..., maxdepth=..., args_template=..., nargs=0, args=0x0) at bytecode.c:641
#20 0x0000000000643cb9 in funcall_lambda (fun=..., nargs=2, arg_vector=0x58b6d9d)
    at eval.c:3021
#21 0x0000000000643418 in apply_lambda (fun=..., args=..., count=22) at eval.c:2880
#22 0x00000000006416da in eval_sub (form=...) at eval.c:2264
#23 0x000000000063c0c3 in Fif (args=...) at eval.c:403
#24 0x0000000000641126 in eval_sub (form=...) at eval.c:2172
#25 0x000000000063ee5d in internal_lisp_condition_case (var=..., bodyform=..., handlers=...) at eval.c:1295
#26 0x000000000063e912 in Fcondition_case (args=...) at eval.c:1221
#27 0x0000000000641126 in eval_sub (form=...) at eval.c:2172
#28 0x000000000063c394 in Fprogn (body=...) at eval.c:449
#29 0x000000000063dff6 in Flet (args=...) at eval.c:963
#30 0x0000000000641126 in eval_sub (form=...) at eval.c:2172
#31 0x000000000063c394 in Fprogn (body=...) at eval.c:449
#32 0x0000000000643c19 in funcall_lambda (fun=..., nargs=1, arg_vector=0x0)
    at eval.c:3014
#33 0x0000000000642d10 in Ffuncall (nargs=2, args=0x7fffe598adf0) at eval.c:2757
#34 0x0000000000642577 in call1 (fn=..., arg1=...) at eval.c:2605
#35 0x00000000006501f6 in mapcar1 (leni=1, vals=0x7fffe598aea0, fn=..., seq=...)
    at fns.c:2471
#36 0x0000000000650615 in Fmapcar (function=..., sequence=...) at fns.c:2523
#37 0x000000000064150d in eval_sub (form=...) at eval.c:2222
#38 0x000000000063c394 in Fprogn (body=...) at eval.c:449
#39 0x000000000063e450 in internal_catch (tag=..., func=0x63c33d <Fprogn>, arg=...)
    at eval.c:1091
#40 0x000000000063e404 in Fcatch (args=...) at eval.c:1068
#41 0x0000000000641126 in eval_sub (form=...) at eval.c:2172
#42 0x000000000064129a in eval_sub (form=...) at eval.c:2184
#43 0x000000000063c394 in Fprogn (body=...) at eval.c:449
#44 0x000000000063dff6 in Flet (args=...) at eval.c:963
#45 0x0000000000641126 in eval_sub (form=...) at eval.c:2172
#46 0x000000000063c394 in Fprogn (body=...) at eval.c:449
#47 0x000000000062cb7b in Fsave_excursion (args=...) at editfns.c:1053
#48 0x0000000000641126 in eval_sub (form=...) at eval.c:2172
#49 0x000000000063c394 in Fprogn (body=...) at eval.c:449
#50 0x0000000000641126 in eval_sub (form=...) at eval.c:2172
#51 0x000000000063c394 in Fprogn (body=...) at eval.c:449
#52 0x0000000000643c19 in funcall_lambda (fun=..., nargs=4, arg_vector=0x0)
    at eval.c:3014
#53 0x0000000000642d10 in Ffuncall (nargs=5, args=0x7fffe598bb50) at eval.c:2757
#54 0x0000000000641eb0 in Fapply (nargs=3, args=0x7fffe598bd80) at eval.c:2374
#55 0x0000000000642f81 in funcall_subr (subr=0xd719a8 <Sapply>, numargs=3, args=0x7fffe598bd80) at eval.c:2798
#56 0x0000000000642bda in Ffuncall (nargs=4, args=0x7fffe598bd78) at eval.c:2743
#57 0x0000000000691dd2 in exec_byte_code (bytestr=..., vector=..., maxdepth=..., args_template=..., nargs=4, args=0x7fffe598c488) at bytecode.c:641
#58 0x00000000006437e1 in funcall_lambda (fun=..., nargs=4, arg_vector=0x7fffe598c480) at eval.c:2943
#59 0x0000000000643418 in apply_lambda (fun=..., args=..., count=4) at eval.c:2880
#60 0x00000000006416da in eval_sub (form=...) at eval.c:2264
#61 0x000000000064129a in eval_sub (form=...) at eval.c:2184
#62 0x000000000063c5d8 in Fsetq (args=...) at eval.c:511
#63 0x0000000000641126 in eval_sub (form=...) at eval.c:2172
#64 0x000000000063c394 in Fprogn (body=...) at eval.c:449
#65 0x0000000000643c19 in funcall_lambda (fun=..., nargs=0, arg_vector=0x0)
    at eval.c:3014
#66 0x0000000000642d10 in Ffuncall (nargs=1, args=0x6117b30) at eval.c:2757
#67 0x00000000006cfe09 in invoke_thread_function () at thread.c:669
#68 0x000000000063eef9 in internal_condition_case (bfun=0x6cfde1 <invoke_thread_function>, handlers=..., hfun=0x6cfe28 <record_thread_error>) at eval.c:1324
#69 0x00000000006cff5c in run_thread (state=0x6117b10) at thread.c:708
#70 0x00007ffff01c52e7 in start_thread () at /usr/lib/libpthread.so.0
#71 0x00007fffefad454f in clone () at /usr/lib/libc.so.6

Lisp Backtrace:
"accept-process-output" (0xe5987f10)
"nnheader-accept-process-output" (0xe5988640)
"nnimap-wait-for-response" (0xe5988e60)
"nnimap-get-response" (0xe5989560)
"nnimap-command" (0xe5989cb0)
"nnimap-change-group" (0xe598a3c0)
"if" (0xe598a6b0)
"condition-case" (0xe598a940)
"let" (0xe598aba0)
0x61e7760 Lisp type 3
"mapcar" (0xe598afa0)
"catch" (0xe598b1f0)
"apply" (0xe598b360)
"let" (0xe598b5e0)
"save-excursion" (0xe598b7a0)
"progn" (0xe598b920)
0x660b320 Lisp type 3
"apply" (0xe598bd80)
"nnir-run-search" (0xe598c480)
"vconcat" (0xe598c760)
"setq" (0xe598c930)
0x61e9790 Lisp type 3

Re: make-thread with lambda form instead of function symbol

[Noam Postavsky]

On Sun, Apr 16, 2017 at 9:12 PM, Eric Abrahamsen
<eric@ericabrahamsen.net> wrote:
>
> (defun nnir-run-query (specs)
>   (let* ((results [])
>          (threads
>           (mapcar
>            (lambda (x)
>              (let* ((server (car x))
>                     (search-engine (nnir-server-to-search-engine server)))
>                (make-thread
>                 (lambda ()
>                   (setq results
>                         (vconcat
>                          (nnir-run-search
>                           search-engine
>                           server
>                           (cdr (assq 'nnir-query-spec specs))
>                           (cadr x))
>                          results))))))
>            (cdr (assq 'nnir-group-spec specs)))))
>     (mapc #'thread-join threads)
>     results))
>
> I'm testing with a single IMAP server query (ie, only one thread is
> being created). The nnir-run-search function for the IMAP backend ends
> up calling nnimap-send-command--> nnimap-wait-for-response-->
> nnheader-accept-process-output--> accept-process-output, so I'm
> expecting that's where the thread yields.
>
> I'm attaching the traceback here. Dunno if it's something I've done
> wrong...

> #4  0x000000000069f953 in Faccept_process_output (process=..., seconds=..., millisec=..., just_this_one=...) at process.c:4566

Is line process.c:4566 this one?

    error ("Attempt to accept output from process %s locked to thread %s",
           SDATA (proc->name), SDATA (XTHREAD (proc->thread)->name));

I guess that indicates you might be doing something wrong (though
obviously Emacs shouldn't be crashing)

Re: make-thread with lambda form instead of function symbol

[Eric Abrahamsen]

Noam Postavsky <npostavs@users.sourceforge.net> writes:

> On Sun, Apr 16, 2017 at 9:12 PM, Eric Abrahamsen
> <eric@ericabrahamsen.net> wrote:
>>
>> (defun nnir-run-query (specs)
>>   (let* ((results [])
>>          (threads
>>           (mapcar
>>            (lambda (x)
>>              (let* ((server (car x))
>>                     (search-engine (nnir-server-to-search-engine server)))
>>                (make-thread
>>                 (lambda ()
>>                   (setq results
>>                         (vconcat
>>                          (nnir-run-search
>>                           search-engine
>>                           server
>>                           (cdr (assq 'nnir-query-spec specs))
>>                           (cadr x))
>>                          results))))))
>>            (cdr (assq 'nnir-group-spec specs)))))
>>     (mapc #'thread-join threads)
>>     results))
>>
>> I'm testing with a single IMAP server query (ie, only one thread is
>> being created). The nnir-run-search function for the IMAP backend ends
>> up calling nnimap-send-command--> nnimap-wait-for-response-->
>> nnheader-accept-process-output--> accept-process-output, so I'm
>> expecting that's where the thread yields.
>>
>> I'm attaching the traceback here. Dunno if it's something I've done
>> wrong...
>
>> #4  0x000000000069f953 in Faccept_process_output (process=..., seconds=..., millisec=..., just_this_one=...) at process.c:4566
>
> Is line process.c:4566 this one?
>
>     error ("Attempt to accept output from process %s locked to thread %s",
>            SDATA (proc->name), SDATA (XTHREAD (proc->thread)->name));
>
> I guess that indicates you might be doing something wrong (though
> obviously Emacs shouldn't be crashing)

So perhaps an existing process can't be "moved" to another thread?
Tomorrow I'll set up a notmuch-indexed server and try that -- in that
case the process should be invoked and concluded within a single
make-thread call, so maybe it won't raise this error.

But the segfault here seems to be coming from a borked attempt to
represent either the process or the thread as a string....

Re: make-thread with lambda form instead of function symbol

[Eli Zaretskii]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Date: Sun, 16 Apr 2017 18:12:17 -0700
> 
> #0  0x000000000058879c in terminate_due_to_signal (sig=6, backtrace_limit=2147483647) at emacs.c:363
> #1  0x000000000061dbf4 in die (msg=0x74e3fd "STRINGP (a)", file=0x74e330 "lisp.h", line=1276) at alloc.c:7339
> #2  0x0000000000582dfd in XSTRING (a=...) at lisp.h:1276
> #3  0x0000000000582e44 in SDATA (string=...) at lisp.h:1327
> #4  0x000000000069f953 in Faccept_process_output (process=..., seconds=..., millisec=..., just_this_one=...) at process.c:4566

Please try the latest master, I hope I fixed this blunder.

In the future, when reporting backtraces from multithreaded Lisp
programs, please always say "thread apply all bt", to produce
backtraces of all the threads, not just the current one.  Otherwise,
some important information could be missing.

Thanks.

Re: make-thread with lambda form instead of function symbol

[Andrew Cohen]

It would be great to get the searches done concurrently. This is the
last serious (IMHO) problem with gnus searches---the searching part can
take a long time. The current implementation tries to collect everything
to minimize connections to the backends (i.e. searching multiple groups
on a single backend should use a single connection) but even imap
searching gets to be a pain when several imap servers are involved.

Just a side comment: the existing nnir-run-query handles multiple groups
with different backends mostly just fine (albeit searching sequentially
rather than concurrently). The limitation is that the search query must
be common to the different backends (this is the big issue that your
general search query language would fix). I (used to) routinely combine
gmane, namazu, and imap groups in my searches (used to since I stopped
using namazu long ago and gmane search is now defunct :().

And a side-side comment: if the different backends allow different
criteria then search groups from different backends will prompt for
different criteria for each backend. Cumbersome but occasionally
helpful.

Andy

Re: make-thread with lambda form instead of function symbol

[Eli Zaretskii]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Date: Sun, 16 Apr 2017 20:45:19 -0700
> 
> >> (defun nnir-run-query (specs)
> >>   (let* ((results [])
> >>          (threads
> >>           (mapcar
> >>            (lambda (x)
> >>              (let* ((server (car x))
> >>                     (search-engine (nnir-server-to-search-engine server)))
> >>                (make-thread
> >>                 (lambda ()
> >>                   (setq results
> >>                         (vconcat
> >>                          (nnir-run-search
> >>                           search-engine
> >>                           server
> >>                           (cdr (assq 'nnir-query-spec specs))
> >>                           (cadr x))
> >>                          results))))))
> >>            (cdr (assq 'nnir-group-spec specs)))))
> >>     (mapc #'thread-join threads)
> >>     results))
> >>
> >> I'm testing with a single IMAP server query (ie, only one thread is
> >> being created). The nnir-run-search function for the IMAP backend ends
> >> up calling nnimap-send-command--> nnimap-wait-for-response-->
> >> nnheader-accept-process-output--> accept-process-output, so I'm
> >> expecting that's where the thread yields.

In this chain of calls, where's the backend process started?

> >     error ("Attempt to accept output from process %s locked to thread %s",
> >            SDATA (proc->name), SDATA (XTHREAD (proc->thread)->name));
> >
> > I guess that indicates you might be doing something wrong (though
> > obviously Emacs shouldn't be crashing)
> 
> So perhaps an existing process can't be "moved" to another thread?

What does it mean "moved" in this context?  How did you try to "move"
a process to another thread?

The ELisp manual says:

  ... by default a process is locked to the thread
  that created it.  When a process is locked to a thread, output from the
  process can only be accepted by that thread.

If you want to be able to accept process output from a thread other
than the one which created it, you need to call set-process-thread.

> But the segfault here seems to be coming from a borked attempt to
> represent either the process or the thread as a string....

Yes.  Should be fixed now.  But my advice is to always give your
thread meaningful names, as doing so will help you with debugging
(unless you are very good in remembering the hex addresses of your
thread objects ;-).

Re: make-thread with lambda form instead of function symbol

[Eric Abrahamsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Date: Sun, 16 Apr 2017 20:45:19 -0700
>>
>> >> (defun nnir-run-query (specs)
>> >>   (let* ((results [])
>> >>          (threads
>> >>           (mapcar
>> >>            (lambda (x)
>> >>              (let* ((server (car x))
>> >>                     (search-engine (nnir-server-to-search-engine server)))
>> >>                (make-thread
>> >>                 (lambda ()
>> >>                   (setq results
>> >>                         (vconcat
>> >>                          (nnir-run-search
>> >>                           search-engine
>> >>                           server
>> >>                           (cdr (assq 'nnir-query-spec specs))
>> >>                           (cadr x))
>> >>                          results))))))
>> >>            (cdr (assq 'nnir-group-spec specs)))))
>> >>     (mapc #'thread-join threads)
>> >>     results))
>> >>
>> >> I'm testing with a single IMAP server query (ie, only one thread is
>> >> being created). The nnir-run-search function for the IMAP backend ends
>> >> up calling nnimap-send-command--> nnimap-wait-for-response-->
>> >> nnheader-accept-process-output--> accept-process-output, so I'm
>> >> expecting that's where the thread yields.
>
> In this chain of calls, where's the backend process started?
>
>> >     error ("Attempt to accept output from process %s locked to thread %s",
>> >            SDATA (proc->name), SDATA (XTHREAD (proc->thread)->name));
>> >
>> > I guess that indicates you might be doing something wrong (though
>> > obviously Emacs shouldn't be crashing)
>>
>> So perhaps an existing process can't be "moved" to another thread?
>
> What does it mean "moved" in this context?  How did you try to "move"
> a process to another thread?
>
> The ELisp manual says:
>
>   ... by default a process is locked to the thread
>   that created it.  When a process is locked to a thread, output from the
>   process can only be accepted by that thread.
>
> If you want to be able to accept process output from a thread other
> than the one which created it, you need to call set-process-thread.

Okay that certainly explains it -- I hadn't found that part of the
manual. The IMAP process in question was long-running, created well
before this thread was, so I was doing exactly the wrong thing. (All I
meant by "move" was starting the process in one thread, and accepting
its output in another.)

>> But the segfault here seems to be coming from a borked attempt to
>> represent either the process or the thread as a string....
>
> Yes.  Should be fixed now.  But my advice is to always give your
> thread meaningful names, as doing so will help you with debugging
> (unless you are very good in remembering the hex addresses of your
> thread objects ;-).

Segfault's gone, thank you. Now to figure out where to stick the call to
`set-process-thread'.

And thanks for the "thread apply all bt" trick, I'll remember that.

Eric

Re: make-thread with lambda form instead of function symbol

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Eli Zaretskii <eliz@gnu.org> writes:

[...]

>> What does it mean "moved" in this context?  How did you try to "move"
>> a process to another thread?
>>
>> The ELisp manual says:
>>
>>   ... by default a process is locked to the thread
>>   that created it.  When a process is locked to a thread, output from the
>>   process can only be accepted by that thread.
>>
>> If you want to be able to accept process output from a thread other
>> than the one which created it, you need to call set-process-thread.
>
> Okay that certainly explains it -- I hadn't found that part of the
> manual. The IMAP process in question was long-running, created well
> before this thread was, so I was doing exactly the wrong thing. (All I
> meant by "move" was starting the process in one thread, and accepting
> its output in another.)
>
> Segfault's gone, thank you. Now to figure out where to stick the call to
> `set-process-thread'.

I tried unlocking all IMAP processes early on, when they're first
created, and so far it seems to be working great!

Re: make-thread with lambda form instead of function symbol

[Eric Abrahamsen]

Andrew Cohen <cohen@bu.edu> writes:

> It would be great to get the searches done concurrently. This is the
> last serious (IMHO) problem with gnus searches---the searching part can
> take a long time. The current implementation tries to collect everything
> to minimize connections to the backends (i.e. searching multiple groups
> on a single backend should use a single connection) but even imap
> searching gets to be a pain when several imap servers are involved.

Subjectively, the thread trick seems to really speed things up. I've
also been looking at some of the IMAP extensions to improve
single-server search -- MULTISEARCH could speed things up significantly,
and wouldn't be hard to implement. I'm not sure about SEARCHRES. It
doesn't look like it actually saves much time, and would be very hard to
integrate with the structure of Gnus searches.

> Just a side comment: the existing nnir-run-query handles multiple groups
> with different backends mostly just fine (albeit searching sequentially
> rather than concurrently). The limitation is that the search query must
> be common to the different backends (this is the big issue that your
> general search query language would fix). I (used to) routinely combine
> gmane, namazu, and imap groups in my searches (used to since I stopped
> using namazu long ago and gmane search is now defunct :().
>
> And a side-side comment: if the different backends allow different
> criteria then search groups from different backends will prompt for
> different criteria for each backend. Cumbersome but occasionally
> helpful.

Actually, in the implementation I'm working on now, I've removed the
additional criteria. The general query language makes selecting imap
keys unnecessary, the gmane "author" criteria can simply be transformed
from the "from" keyword (not to mention that this support is now
theoretical!), and all the other backend criteria specify groups: can't
we just take that from the group search spec?

I'll be happy to re-instate criteria if they really turn out to be
necessary, but I'd like to try to do away with them if possible.

Eric

Re: make-thread with lambda form instead of function symbol

[Andrew Cohen]

>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:

    >> And a side-side comment: if the different backends allow
    >> different criteria then search groups from different backends
    >> will prompt for different criteria for each backend. Cumbersome
    >> but occasionally helpful.

    Eric> Actually, in the implementation I'm working on now, I've
    Eric> removed the additional criteria. The general query language
    Eric> makes selecting imap keys unnecessary, the gmane "author"
    Eric> criteria can simply be transformed from the "from" keyword
    Eric> (not to mention that this support is now theoretical!), and
    Eric> all the other backend criteria specify groups: can't we just
    Eric> take that from the group search spec?

Eliminating the criteria sounds good---it was present just to deal with
the non-uniformity of search query formats across different  backends,
so once the universal language is in place it should go away forever.

Best,
Andy

Re: make-thread with lambda form instead of function symbol

[Eric Abrahamsen]

Andrew Cohen <cohen@bu.edu> writes:

>>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>     >> And a side-side comment: if the different backends allow
>     >> different criteria then search groups from different backends
>     >> will prompt for different criteria for each backend. Cumbersome
>     >> but occasionally helpful.
>
>     Eric> Actually, in the implementation I'm working on now, I've
>     Eric> removed the additional criteria. The general query language
>     Eric> makes selecting imap keys unnecessary, the gmane "author"
>     Eric> criteria can simply be transformed from the "from" keyword
>     Eric> (not to mention that this support is now theoretical!), and
>     Eric> all the other backend criteria specify groups: can't we just
>     Eric> take that from the group search spec?
>
> Eliminating the criteria sounds good---it was present just to deal with
> the non-uniformity of search query formats across different  backends,
> so once the universal language is in place it should go away forever.

Excellent, that's good to hear. I'll keep working on this. Oddly, the
main slowdown now is simply finding good documentation on the various
engines' search syntax. I'll also see if there are any other search
engines worth adding.

Re: make-thread with lambda form instead of function symbol

[Eric Abrahamsen]

Andrew Cohen <cohen@bu.edu> writes:

> It would be great to get the searches done concurrently. This is the
> last serious (IMHO) problem with gnus searches---the searching part can
> take a long time. The current implementation tries to collect everything
> to minimize connections to the backends (i.e. searching multiple groups
> on a single backend should use a single connection) but even imap
> searching gets to be a pain when several imap servers are involved.

Bah, this isn't quite working. Searching more than one IMAP server
causes weird results and/or hangs. Searching other backends, or one IMAP
server plus other backends, seems to work okay. I have a hunch that this
comes from the way Gnus backends share the nntp-server-buffer variable
and swap symbols around -- I think the processes might be stepping on
each others' toes from thread to thread. nnimap-wait-for-response loops
accept-process-output on a 0.01 second timeout, so I can imagine that
two or more threads could be fighting over the meaning of
nntp-server-buffer.

Anyway, I've gone down enough garden paths here. I'm going to leave
things set up for threading, but not actually use threads for now. Once
the rest of the stuff is in place there will be time to figure out
what's going wrong.

Eric