guile and coroutines controlled from C

guile and coroutines controlled from C

[Vincent Bernat]

Hi!

I would like to add Guile scripting to an actual program to allow a user
to write simple network-related scenarios. Those scenarios will be run
in parallel on several entities. The application is event-driven and I
want to hide this fact to the user. She should be able to write a
scenario in a simple blocking style:

#v+
(let ((v1 (do-something-blocking "arg1" "arg2"))
      (v2 (do-something-blocking "arg3" "arg4")))
 (when (< (+ v1 v2) 10) (...)))
#v-

`do-something-blocking` is a registered function written in C. I want to
be able to pause the execution inside this function and then resume it
later (when I my program got the data wanted by the user). In the
meantime, another "cooperative thread" may be executed.

In Lua, this is something that can be done with coroutines which can be
yield and resumed from C. In Guile, there are continuations but I have
hard time to understand how this should work from C.

What would be the equivalent of this in Guile?
 http://kristianrumberg.wordpress.com/2010/11/21/135/
-- 
 /*
  * We used to try various strange things. Let's not.
  */
	2.2.16 /usr/src/linux/fs/buffer.c

Re: guile and coroutines controlled from C

[Ian Price]

Vincent Bernat <bernat@luffy.cx> writes:

> Hi!
> I would like to add Guile scripting to an actual program to allow a user
> to write simple network-related scenarios. Those scenarios will be run
> in parallel on several entities. The application is event-driven and I
> want to hide this fact to the user. She should be able to write a
> scenario in a simple blocking style:

Hi, always happy to new uses for guile.
>
> #v+
> (let ((v1 (do-something-blocking "arg1" "arg2"))
>       (v2 (do-something-blocking "arg3" "arg4")))
>  (when (< (+ v1 v2) 10) (...)))
> #v-
>
> `do-something-blocking` is a registered function written in C. I want to
> be able to pause the execution inside this function and then resume it
> later (when I my program got the data wanted by the user). In the
> meantime, another "cooperative thread" may be executed.
>
> In Lua, this is something that can be done with coroutines which can be
> yield and resumed from C. In Guile, there are continuations but I have
> hard time to understand how this should work from C.
>
> What would be the equivalent of this in Guile?
>  http://kristianrumberg.wordpress.com/2010/11/21/135/

So, is it a C function you want to pause and resume, or a scheme one?
Your first statement implies the former, but that example you pointed to
implies you want scheme coroutines, that are callable from C. Both
should be possible, but I'd need to do some experimentation before
providing help with the former, as I'm not entirely sure about the
interaction C and continuations in the vm.

As for scheme coroutines, I think Wingo does that in one of the
asynchronous IO branches, but I'm not sure if it exported for general
use.

A quick hack, I wrote is posted below. Threads are represented by thunks
and are ran by calling said thunk. They would be called from C in the
same way as any other function. In practice, it would be cleaner to
create a new record type for these.

(use-modules (ice-9 q)) ;; yech, but it'll do

(define (list->q l)
  (define q (make-q))
  (for-each (lambda (x) (enq! q x)) l)
  q)

(define (run threads)
  ;; simple round robin scheduler
  (define *q* (list->q threads))
  (while (not (q-empty? *q*))
    (let ((thread (deq! *q*)))
      (when (eq? 'not-done (thread))
        (enq! *q* thread)))))

(define (make-thread thread)
  ;; NB. this might be cleaner with delimited continuations, but they
  ;; seem to scare people even more than continuations for some reason...
  
  ;; A thread is a thunk, that performs some computation, then returns
  ;; either 'not-done or 'done
  (lambda ()
    (call-with-current-continuation
     (lambda (start) ;; capture "the stack" before routines is ran
       (thread (lambda ()
                 (call-with-current-continuation
                  (lambda (current)
                    ;; save current "stack" for thread
                    ;; so that we continue from there next time
                    (set! thread current)
                    ;; then we need to cleanup, and go back to where
                    ;; we were before the thread began executing
                    (start 'not-done)))))
       ;; should probably mutate thread one last time, so that further
       ;; calls always just return 'done
       'done))))

(define thread1
  (make-thread
   (lambda (yield)
     (let loop ((i 3))
       (if (zero? i)
           'done
           (begin
             (display "in thread 1...\n")
             (yield)
             (loop (- i 1))))))))

(define thread2
  (make-thread
   (lambda (yield)
     (let loop ((i 5))
       (if (zero? i)
           'done
           (begin
             (display "in thread 2...\n")
             (yield)
             (loop (- i 1))))))))

(define thread3
  (make-thread
   (lambda (yield)
     (let loop ((i 4))
       (if (zero? i)
           'done
           (begin
             (display "in thread 3...\n")
             (yield)
             (loop (- i 1))))))))

(run (list thread1 thread2 thread3))
;; Strangely enough, I was able to write this out and have it work first
;; time, which is a first for me writing continuation code :P


~ $ guile -s /tmp/contexample.scm 
in thread 1...
in thread 2...
in thread 3...
in thread 1...
in thread 2...
in thread 3...
in thread 1...
in thread 2...
in thread 3...
in thread 2...
in thread 3...
in thread 2...
~ $ 

I hope that gives you a head start. If I've confused you more, I
can only apologise. If you aren't familiar with continuations at all, 
you could try reading http://tmp.barzilay.org/cont.txt

-- 
Ian Price

"Programming is like pinball. The reward for doing it well is
the opportunity to do it again" - from "The Wizardy Compiled"

Re: guile and coroutines controlled from C

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 659 bytes --]

() Ian Price <ianprice90@googlemail.com>
() Mon, 30 Jul 2012 05:56:43 +0100

   (use-modules (ice-9 q)) ;; yech, but it'll do

   [...]

See also module (ice-9 runq), documented at:

 http://www.gnuvola.org/software/guile/doc/Run-Queues.html

and possibly other places.

-- 
Thien-Thi Nguyen ..................................... GPG key: 4C807502
.                  NB: ttn at glug dot org is not me                   .
.                 (and has not been since 2007 or so)                  .
.                        ACCEPT NO SUBSTITUTES                         .
........... please send technical questions to mailing lists ...........

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: guile and coroutines controlled from C

[Vincent Bernat]

 ❦ 30 juillet 2012 06:56 CEST, Ian Price <ianprice90@googlemail.com> :

>> #v+
>> (let ((v1 (do-something-blocking "arg1" "arg2"))
>>       (v2 (do-something-blocking "arg3" "arg4")))
>>  (when (< (+ v1 v2) 10) (...)))
>> #v-
>>
>> `do-something-blocking` is a registered function written in C. I want to
>> be able to pause the execution inside this function and then resume it
>> later (when I my program got the data wanted by the user). In the
>> meantime, another "cooperative thread" may be executed.
>>
>> In Lua, this is something that can be done with coroutines which can be
>> yield and resumed from C. In Guile, there are continuations but I have
>> hard time to understand how this should work from C.
>>
>> What would be the equivalent of this in Guile?
>>  http://kristianrumberg.wordpress.com/2010/11/21/135/
>
> So, is it a C function you want to pause and resume, or a scheme one?
> Your first statement implies the former, but that example you pointed to
> implies you want scheme coroutines, that are callable from C. Both
> should be possible, but I'd need to do some experimentation before
> providing help with the former, as I'm not entirely sure about the
> interaction C and continuations in the vm.

Hum. Both I think. :) I want a user to provide a scheme function that
will be run several times in parallel against dozens of targets with
cooperative multithreading. The user does not have to know how
cooperation works. It just uses some primitives (implemented in C) that
may yield control to another thread. The C application is event-driven.

#v+
(grab-something-from-network device arg1 arg2)
(do-some-computation-with-result)
(grab-something-else-from-network device arg3 arg4)
(grab-something-else device arg5)
(some-computation)
#v-

The `grab-something-*` functions will be implemented in C. They will
pause the current thread and resume it when data get available. In
Python, there is something called "greenlets" that use the same concept.

> As for scheme coroutines, I think Wingo does that in one of the
> asynchronous IO branches, but I'm not sure if it exported for general
> use.
>
> A quick hack, I wrote is posted below. Threads are represented by thunks
> and are ran by calling said thunk. They would be called from C in the
> same way as any other function. In practice, it would be cleaner to
> create a new record type for these.
[...]

Your example is great! It fits exactly what I need. Since I am still
pretty new, I need some time to understand each line but this seems a
very good start for what I want to do. I just need to translate some
parts in C since the event loop is here. From what I understand, only
the `run` function needs to be moved in C. The remaining of the code can
be hidden in functions or macros (but it would help if the `yield` part
could be translated in C).

Just one other question.

> (define thread1
>   (make-thread
>    (lambda (yield)
>      (let loop ((i 3))
>        (if (zero? i)
>            'done
>            (begin
>              (display "in thread 1...\n")
>              (yield)
>              (loop (- i 1))))))))

After `yield`, the routine is resumed when scheduled by the previous
`run` function. In my case, yielding happened because I was waiting for data
from the network. How could I return the appropriate data on resume? I
could call some function that will provide the revelant data from some
cache just after yield:

#v+
(yield)
(some-computation-with (grab-result-local-to-this-thread))
#v-

However, maybe `yield` could return the result?

> I hope that gives you a head start. If I've confused you more, I
> can only apologise. If you aren't familiar with continuations at all, 
> you could try reading http://tmp.barzilay.org/cont.txt

Thanks for your help. I am still a bit confused but less than a day
before. I just need some time to experiment a bit.
-- 
panic("Aarggh: attempting to free lock with active wait queue - shoot Andy");
	2.0.38 /usr/src/linux/fs/locks.c

Re: guile and coroutines controlled from C

[Ian Price]

Vincent Bernat <bernat@luffy.cx> writes:

> Your example is great! It fits exactly what I need. Since I am still
> pretty new, I need some time to understand each line but this seems a
> very good start for what I want to do. I just need to translate some
> parts in C since the event loop is here. From what I understand, only
> the `run` function needs to be moved in C. The remaining of the code can
> be hidden in functions or macros (but it would help if the `yield` part
> could be translated in C).
Well, it all depends on the exact API you want to provide/code to. So,
let's take your "do-something-blocking" function. One way to do this is
to have the yield function be a piece of local state that is only
available when the "thread" is running, for that I'd use a parameter, or
fluid variable. (see the sections 6.21.7/6.21.8 of the manual). Then
these functions would query that, and call the yield function to return,
after registering that the work needs to be done in a different thread
(or however you plan on doing this). I'd want the thread to be known to
the function, so that it could store, say store the return value in it,
so that you don't need to explicitly provide this when you resume. You
probably also want to provide a flag in the thread data type saying
whether or not it could be resumed.

Like I say, it all depends on the API you want to code to. I could whip
up some example code that does the above if you'd like.

> Just one other question.
>
>> (define thread1
>>   (make-thread
>>    (lambda (yield)
>>      (let loop ((i 3))
>>        (if (zero? i)
>>            'done
>>            (begin
>>              (display "in thread 1...\n")
>>              (yield)
>>              (loop (- i 1))))))))
>
> After `yield`, the routine is resumed when scheduled by the previous
> `run` function. In my case, yielding happened because I was waiting for data
> from the network. How could I return the appropriate data on resume? I
> could call some function that will provide the revelant data from some
> cache just after yield:
>
> #v+
> (yield)
> (some-computation-with (grab-result-local-to-this-thread))
> #v-
>
> However, maybe `yield` could return the result?
Right, when you invoke a continuation, you actually give it a value,
which is used as the return value for the call. I actually make use of
this in the code above, so that the return value of the threads function
returns 'not-done. In this case, when you want a thread to return a
value you just call that thread variable with the value. In the example
I posted, the return value is actually always the yield
function. Though, some rewriting could allow you to return other values.

-- 
Ian Price

"Programming is like pinball. The reward for doing it well is
the opportunity to do it again" - from "The Wizardy Compiled"

Re: guile and coroutines controlled from C

[Ian Price]

Thien-Thi Nguyen <ttn@gnuvola.org> writes:

> () Ian Price <ianprice90@googlemail.com>
> () Mon, 30 Jul 2012 05:56:43 +0100
>
>    (use-modules (ice-9 q)) ;; yech, but it'll do
>
>    [...]
>
> See also module (ice-9 runq), documented at:
>
>  http://www.gnuvola.org/software/guile/doc/Run-Queues.html
>
> and possibly other places.

Cool, I was not aware of this, it may actually something similar to what
Vincent wants. Though, I was just lamenting the specifics of the (ice-9
q) module which I am not a fan of api-wise.

-- 
Ian Price

"Programming is like pinball. The reward for doing it well is
the opportunity to do it again" - from "The Wizardy Compiled"

Re: guile and coroutines controlled from C

[Vincent Bernat]

 ❦  1 août 2012 15:13 CEST, Ian Price <ianprice90@googlemail.com> :

>> Your example is great! It fits exactly what I need. Since I am still
>> pretty new, I need some time to understand each line but this seems a
>> very good start for what I want to do. I just need to translate some
>> parts in C since the event loop is here. From what I understand, only
>> the `run` function needs to be moved in C. The remaining of the code can
>> be hidden in functions or macros (but it would help if the `yield` part
>> could be translated in C).
> Well, it all depends on the exact API you want to provide/code to. So,
[...]

Ian,

Thanks for your replies! Now, I just need some time to process all
that. I'll come back here once I am done (with some code I hope)!
-- 
panic("Attempted to kill the idle task!");
	2.2.16 /usr/src/linux/kernel/exit.c