* Re: Guile fibers return values @ 2020-01-05 1:30 Zelphir Kaltstahl 2020-01-05 12:33 ` Chris Vine 0 siblings, 1 reply; 16+ messages in thread From: Zelphir Kaltstahl @ 2020-01-05 1:30 UTC (permalink / raw) To: guile-user Hello Guile Users, so I figured out an example for using channels, but I am not sure, if that is the only way to get results from a fiber: ~~~~8<~~~~8<~~~~ (use-modules (fibers) (fibers channels) (ice-9 match)) ;; Define a procedure to run in a fiber. (define fiber1-proc (lambda (in-chan out-chan) ;; Look for mesages. (let loop ([received-proc (lambda (data) 'no-proc-received)]) (match ;; Write arguments to the current output port, and return the last ;; argument. This will get the message received and write to current ;; output port an information, that the get-message procedure was ;; called. (pk 'fiber1-proc-called-get-message (get-message in-chan)) ;; Match anything tagged as procedure and store it in the argument for ;; the named let. [('proc . proc) (loop proc)] ;; Match anything labeled as data and apply the stored procedure to ;; it. If no procedure has been received yet, use the default one. [('data . data) (put-message out-chan (received-proc data)) ;; Loop again with the default procedure, awaiting a new procedure and ;; data for it. (loop (lambda (data) 'no-proc-received))] ;; Have a default reaction to anything, but the correctly tagged ;; messages. [any-other-message (put-message out-chan 'unrecognized-message) ;; Allow for unrecognized messages in between correct communication. (loop received-proc)]) ;; Continue looking for messages. (loop received-proc)))) (run-fibers (lambda () (let ((fiber1-in-chan (make-channel)) (fiber1-out-chan (make-channel))) ;; Spawn a fiber to run fiber1-proc, which internally looks for messages on ;; its in-channel. (spawn-fiber (lambda () (fiber1-proc fiber1-in-chan fiber1-out-chan))) ;; Send a mssage to the fiber. (put-message fiber1-in-chan ;; Send some tagged data, in this case the procedure to use. (cons 'proc ;; A procedure, which checks all things in data for ;; whether they are even numbers and builds a list of ;; the answers. (lambda (data) (let loop ([remaining-data data]) (cond [(null? remaining-data) '()] [else (cons (even? (car remaining-data)) (loop (cdr remaining-data)))]))))) ;; Then put the data on the channel. (put-message fiber1-in-chan (cons 'data '(0 1 2 3 4 5 6 7 8 9))) ;; Look for the answer on the out-channel of the fiber. (display (simple-format #f "~a\n" (pk 'main-thread-called-peek (get-message fiber1-out-chan)))) ;; And then do it again. ;; Send a mssage to the fiber. (put-message fiber1-in-chan ;; Send some tagged data, in this case the procedure to use. (cons 'proc ;; A procedure, which checks all things in data for ;; whether they are even numbers and builds a list of ;; the answers. (lambda (data) (let loop ([remaining-data data]) (cond [(null? remaining-data) '()] [else (cons (even? (car remaining-data)) (loop (cdr remaining-data)))]))))) ;; Then put the data on the channel. (put-message fiber1-in-chan (cons 'data '(0 1 2 3 4 5 6 7 8 9))) ;; Look for the answer on the out-channel of the fiber. (display (simple-format #f "~a\n" (pk 'main-thread-called-peek (get-message fiber1-out-chan))))))) ~~~~>8~~~~>8~~~~ Is there another way or anything quite wrong in this example? This way of communication between the fiber and the main process seems in the style of Racket's places. Except that I can send normal procedures / lambdas to the fiber, which is great on a single machine, while I need to send serializable lambdas to Racket places (and I have not gotten to do that yet). Is there a restriction on the kind of lambdas I can send on a channel as I did in the example above? Regards, Zelphir Regards, Zelphir ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Guile fibers return values 2020-01-05 1:30 Guile fibers return values Zelphir Kaltstahl @ 2020-01-05 12:33 ` Chris Vine 2020-01-05 12:58 ` Zelphir Kaltstahl 0 siblings, 1 reply; 16+ messages in thread From: Chris Vine @ 2020-01-05 12:33 UTC (permalink / raw) To: guile-user On Sun, 5 Jan 2020 02:30:06 +0100 Zelphir Kaltstahl <zelphirkaltstahl@posteo.de> wrote: [snip] > This way of communication between the fiber and the main process seems > in the style of Racket's places. Except that I can send normal > procedures / lambdas to the fiber, which is great on a single machine, > while I need to send serializable lambdas to Racket places (and I have > not gotten to do that yet). > > Is there a restriction on the kind of lambdas I can send on a channel as > I did in the example above? I may well be missing your point, mainly because I don't know what you mean by "the main process" - all the fibers are part of the same process, and can be run in the same native thread if you want. run-fibers runs what amounts to a scheduler and does not return until the thunk passed to it returns. So if by "the main proccess" you mean the thunk which is running on a fiber scheduler, then you know it has finished when run-fibers returns, after which you can execute what other non-fiber code you want to execute. run-fibers will return the value (if any) returned by the thunk which it runs. Within the thunk run by run-fibers, you normally synchronize using channels. At it's absolute simplest it can be this: (display (run-fibers (lambda () (let ((channel (make-channel))) (spawn-fiber (lambda () (sleep 1) ;; do some work (put-message channel "hello world"))) (simple-format #t "~a~%" (get-message channel)) "finished\n")))) Here the "main" thunk (the one passed to run-fibers which returns "finished\n") will not finish until the fiber thunk has finished, because of the wait on the channel. If you spawn multiple fibers and the "main" thunk does not wait for the fibers like this, and you therefore need to ensure additionally that run-fibers does not return until all the fiber thunks have finished, you can set the drain argument of run-fibers to #t. Probably in that case your "main" thunk will not return a meaningful value. You say you want "to parallelize some algorithm". If that is your main aim, consider guile's parallel forms of parallel, let-par and friends. Chris ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Guile fibers return values 2020-01-05 12:33 ` Chris Vine @ 2020-01-05 12:58 ` Zelphir Kaltstahl 2020-01-05 14:28 ` Chris Vine [not found] ` <20200105142358.4ad96d15a23a0b947b2d55e3@gmail.com> 0 siblings, 2 replies; 16+ messages in thread From: Zelphir Kaltstahl @ 2020-01-05 12:58 UTC (permalink / raw) To: Chris Vine; +Cc: guile-user Thank you for the detailed explanation! By "process" I meant only "sequence of steps performed", the main thunk in run-fibers, separate from the steps that are run in the spawned fiber, not really OS process or thread. I will take a look again at the parallel forms and think about whether I want to use them or fibers. Originally I had my algorithm in Racket and could not get it to work in parallel, unless I explore places and serializable lambdas more. I think fibers are more flexible than the parallel forms though, as one could also build a pipeline using fibers or any kind of network of connected computation tasks, while the parallel forms split a task immediately and then join again. Not sure any of the additional flexibility of fibers helps me. Perhaps I can use both and abstract from it with an additional abstraction layer. Then my code could also be used more easily in other Schemes. This is my project: https://notabug.org/ZelphirKaltstahl/guile-ml/src/wip-port-to-guile I still am not sure though, if I can simply use any lambda I want and send that to a fiber, or I need to look out for things like "What is in the environment of the lambda?". It would be good to know that. I guess it depends on how data sent on channels is handled in the fibers library. Regards, Zelphir On 1/5/20 1:33 PM, Chris Vine wrote: > On Sun, 5 Jan 2020 02:30:06 +0100 > Zelphir Kaltstahl <zelphirkaltstahl@posteo.de> wrote: > [snip] >> This way of communication between the fiber and the main process seems >> in the style of Racket's places. Except that I can send normal >> procedures / lambdas to the fiber, which is great on a single machine, >> while I need to send serializable lambdas to Racket places (and I have >> not gotten to do that yet). >> >> Is there a restriction on the kind of lambdas I can send on a channel as >> I did in the example above? > I may well be missing your point, mainly because I don't know what you > mean by "the main process" - all the fibers are part of the same > process, and can be run in the same native thread if you want. > > run-fibers runs what amounts to a scheduler and does not return until > the thunk passed to it returns. So if by "the main proccess" you mean > the thunk which is running on a fiber scheduler, then you know it has > finished when run-fibers returns, after which you can execute what > other non-fiber code you want to execute. run-fibers will return the > value (if any) returned by the thunk which it runs. > > Within the thunk run by run-fibers, you normally synchronize using > channels. At it's absolute simplest it can be this: > > (display (run-fibers > (lambda () > (let ((channel (make-channel))) > (spawn-fiber > (lambda () > (sleep 1) ;; do some work > (put-message channel "hello world"))) > (simple-format #t "~a~%" (get-message channel)) > "finished\n")))) > > Here the "main" thunk (the one passed to run-fibers which returns > "finished\n") will not finish until the fiber thunk has finished, > because of the wait on the channel. If you spawn multiple fibers and > the "main" thunk does not wait for the fibers like this, and you > therefore need to ensure additionally that run-fibers does not return > until all the fiber thunks have finished, you can set the drain > argument of run-fibers to #t. Probably in that case your "main" thunk > will not return a meaningful value. > > You say you want "to parallelize some algorithm". If that is your main > aim, consider guile's parallel forms of parallel, let-par and friends. > > Chris > ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Guile fibers return values 2020-01-05 12:58 ` Zelphir Kaltstahl @ 2020-01-05 14:28 ` Chris Vine [not found] ` <20200105142358.4ad96d15a23a0b947b2d55e3@gmail.com> 1 sibling, 0 replies; 16+ messages in thread From: Chris Vine @ 2020-01-05 14:28 UTC (permalink / raw) To: guile-user On Sun, 5 Jan 2020 13:58:24 +0100 Zelphir Kaltstahl <zelphirkaltstahl@posteo.de> wrote: > Thank you for the detailed explanation! > > By "process" I meant only "sequence of steps performed", the main thunk > in run-fibers, separate from the steps that are run in the spawned > fiber, not really OS process or thread. > > I will take a look again at the parallel forms and think about whether I > want to use them or fibers. > > Originally I had my algorithm in Racket and could not get it to work in > parallel, unless I explore places and serializable lambdas more. > > I think fibers are more flexible than the parallel forms though, as one > could also build a pipeline using fibers or any kind of network of > connected computation tasks, while the parallel forms split a task > immediately and then join again. Not sure any of the additional > flexibility of fibers helps me. Perhaps I can use both and abstract from > it with an additional abstraction layer. Then my code could also be used > more easily in other Schemes. Fibers are more flexible than guile's parallel forms as you say (of course, if you have your own thread pool available that can also be more flexible than the parallel forms), but fibers' main other feature is that they will multi-plex i/o operations on the same native thread using guile's suspendable ports and coroutines. Any one native OS thread run by the fibers' scheduler may therefore have more than one coroutine running on it. So, beware having blocking operations when using fibers, because between fibers running on the same native OS thread, concurrency is co-operative - coroutine task switching occurs on a port suspending or the fiber yielding, although there is some work stealing implemented between OS threads which will help you out. (From the blocking point of view, do not use the simple-format procedure as in my toy example because it is not suspendable-port-safe.) If your work is i/o-bound rather than cpu-bound, fibers are likely to be especially useful. > This is my project: > https://notabug.org/ZelphirKaltstahl/guile-ml/src/wip-port-to-guile > > I still am not sure though, if I can simply use any lambda I want and > send that to a fiber, or I need to look out for things like "What is in > the environment of the lambda?". It would be good to know that. I guess > it depends on how data sent on channels is handled in the fibers library. The lambda closures passed to run-fibers and spawn-fiber are like any other lambda closure. In particular, beware of closures with shared mutable state. Apart from that you should be fine as regards closures but I am not convinced that I sufficiently understand your use case to comment further. On data sent through channels, I haven't looked at it but I have always assumed that objects such as lists, vectors, records and so forth which are passed through channels are passed by reference in the ordinary way, so don't mutate them after putting them in a channel if more than one fiber may subsequently have them in use concurrently. The purpose of channels is to provide safe synchronization. Chris ^ permalink raw reply [flat|nested] 16+ messages in thread
[parent not found: <20200105142358.4ad96d15a23a0b947b2d55e3@gmail.com>]
* Re: Guile fibers return values [not found] ` <20200105142358.4ad96d15a23a0b947b2d55e3@gmail.com> @ 2020-01-05 18:22 ` Zelphir Kaltstahl 2020-01-05 21:45 ` Chris Vine 0 siblings, 1 reply; 16+ messages in thread From: Zelphir Kaltstahl @ 2020-01-05 18:22 UTC (permalink / raw) To: Chris Vine; +Cc: Guile User Hi Chris! On 1/5/20 3:23 PM, Chris Vine wrote: > On Sun, 5 Jan 2020 13:58:24 +0100 > Zelphir Kaltstahl <zelphirkaltstahl@posteo.de> wrote: >> Thank you for the detailed explanation! >> >> By "process" I meant only "sequence of steps performed", the main thunk >> in run-fibers, separate from the steps that are run in the spawned >> fiber, not really OS process or thread. >> >> I will take a look again at the parallel forms and think about whether I >> want to use them or fibers. >> >> Originally I had my algorithm in Racket and could not get it to work in >> parallel, unless I explore places and serializable lambdas more. >> >> I think fibers are more flexible than the parallel forms though, as one >> could also build a pipeline using fibers or any kind of network of >> connected computation tasks, while the parallel forms split a task >> immediately and then join again. Not sure any of the additional >> flexibility of fibers helps me. Perhaps I can use both and abstract from >> it with an additional abstraction layer. Then my code could also be used >> more easily in other Schemes. > Fibers are more flexible than guile's parallel forms as you say (of > course, if you have your own thread pool available that can also be > more flexible than the parallel forms), but fibers' main other feature > is that they will multi-plex i/o operations on the same native thread > using guile's suspendable ports and coroutines. Any one native OS > thread run by the fibers' scheduler may therefore have more than one > coroutine running on it. So, beware having blocking operations when > using fibers, because between fibers running on the same native OS > thread, concurrency is co-operative - coroutine task switching occurs on > a port suspending or the fiber yielding, although there is some work > stealing implemented between OS threads which will help you out. (From > the blocking point of view, do not use the simple-format procedure as in > my toy example because it is not suspendable-port-safe.) > > If your work is i/o-bound rather than cpu-bound, fibers are likely to be > especially useful. I think the decision tree calculations, which I want to parallelize, are not I/O related. However, I am not quite sure I understand the whole suspendable port thing, but here is what I think it is: When some I/O happens in a fiber, the fiber is still able to pause (suspend, yield, …) at that point, because the suspendable ports work in such a way, that no harm is done in doing so. Later the I/O work can be resumed (woken up from suspension). This quality had to be built into Guile first, before fibers were able to take advantage of it. Is this correct? But I do not understand, why this is not the case with normal OS threads. Maybe it can be done but is not convenient or difficult to get right, to work with suspendable ports, when not using the fibers library? And why is simple-format not "suspendable-port-safe"? (What does a procedure need to do, in order to be "suspendable-port-safe"?) >> This is my project: >> https://notabug.org/ZelphirKaltstahl/guile-ml/src/wip-port-to-guile >> >> I still am not sure though, if I can simply use any lambda I want and >> send that to a fiber, or I need to look out for things like "What is in >> the environment of the lambda?". It would be good to know that. I guess >> it depends on how data sent on channels is handled in the fibers library. > The lambda closures passed to run-fibers and spawn-fiber are like any > other lambda closure. In particular, beware of closures with shared > mutable state. Apart from that you should be fine as regards closures > but I am not convinced that I sufficiently understand your use case to > comment further. On data sent through channels, I haven't looked at it > but I have always assumed that objects such as lists, vectors, records > and so forth which are passed through channels are passed by reference > in the ordinary way, so don't mutate them after putting them in a > channel if more than one fiber may subsequently have them in use > concurrently. The purpose of channels is to provide safe > synchronization. Good advice, thanks. I was not planning to do mutation on the things received from channels in a fiber, but it is good to remember, that it might be problematic to do so, even with vectors or structs. So I better create new structs or vectors, when creating a return message. With the parallel forms, isn't it the case, that at each call of such a form, new OS threads are created? In this case it might be a good idea to create a fibers scheduler and reuse it, if that is possible, so that I do not need to create my own process pool kind of thingy. Regards, Zelphir ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Guile fibers return values 2020-01-05 18:22 ` Zelphir Kaltstahl @ 2020-01-05 21:45 ` Chris Vine 2020-01-06 19:42 ` Zelphir Kaltstahl 0 siblings, 1 reply; 16+ messages in thread From: Chris Vine @ 2020-01-05 21:45 UTC (permalink / raw) To: Zelphir Kaltstahl; +Cc: Guile User On Sun, 5 Jan 2020 19:22:14 +0100 Zelphir Kaltstahl <zelphirkaltstahl@posteo.de> wrote: > I think the decision tree calculations, which I want to parallelize, are > not I/O related. However, I am not quite sure I understand the whole > suspendable port thing, but here is what I think it is: > > When some I/O happens in a fiber, the fiber is still able to pause > (suspend, yield, …) at that point, because the suspendable ports work in > such a way, that no harm is done in doing so. Later the I/O work can be > resumed (woken up from suspension). This quality had to be built into > Guile first, before fibers were able to take advantage of it. > > Is this correct? Yes, suspendable ports were first implemented in guile-2.2. They are used by 8-sync (https://www.gnu.org/software/8sync/), guile-a-sync2 (https://github.com/ChrisVine/guile-a-sync2/wiki) and fibers, and possibly by other libraries. The basic arrangement is that if a port's file descriptor is not ready, then its continuation is saved, and is resumed by the scheduler when it becomes ready. fibers use epoll rather than POSIX select or poll for this. > But I do not understand, why this is not the case with normal OS > threads. Maybe it can be done but is not convenient or difficult to get > right, to work with suspendable ports, when not using the fibers library? > > And why is simple-format not "suspendable-port-safe"? (What does a > procedure need to do, in order to be "suspendable-port-safe"?) simple-format does not suspend as it is implemented in C in libguile and its continuation is not available to scheme code. There is a list of those of guile's i/o procedures which do (and which do not) suspend here, in the second and third paragraphs, although it does not mention format/simple-format: https://github.com/ChrisVine/guile-a-sync2/wiki/await-ports (That library has nothing to do with fibers, but as mentioned above it happens to use suspendable ports for similar purposes): None of this is likely to be relevant to your use case. [snip] > With the parallel forms, isn't it the case, that at each call of such a > form, new OS threads are created? In this case it might be a good idea > to create a fibers scheduler and reuse it, if that is possible, so that > I do not need to create my own process pool kind of thingy. guile's parallel forms are implemented using futures, which use an internal thread pool according to this: https://www.gnu.org/software/guile/docs/master/guile.html/Futures.html#Futures "Internally, a fixed-size pool of threads is used to evaluate futures, such that offloading the evaluation of an expression to another thread doesn’t incur thread creation costs. By default, the pool contains one thread per available CPU core, minus one, to account for the main thread." Chris ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Guile fibers return values 2020-01-05 21:45 ` Chris Vine @ 2020-01-06 19:42 ` Zelphir Kaltstahl 2020-01-06 21:14 ` Chris Vine 0 siblings, 1 reply; 16+ messages in thread From: Zelphir Kaltstahl @ 2020-01-06 19:42 UTC (permalink / raw) To: Chris Vine; +Cc: Guile User Hey Chris! Thanks for your informative reply! I did not remember, that the parallel forms are already that clever. It might be the case, that I only need to use futures or parallel forms then. I have a question regarding futures though. In Racket the futures have some limitations, where one needs to use a different number type to enable them in some cases to run in parallel – wait, I am looking for the link … here: https://docs.racket-lang.org/guide/parallelism.html – Is there any similar restriction for futures in Guile? Regards, Zelphir On 1/5/20 10:45 PM, Chris Vine wrote: > On Sun, 5 Jan 2020 19:22:14 +0100 > Zelphir Kaltstahl <zelphirkaltstahl@posteo.de> wrote: >> I think the decision tree calculations, which I want to parallelize, are >> not I/O related. However, I am not quite sure I understand the whole >> suspendable port thing, but here is what I think it is: >> >> When some I/O happens in a fiber, the fiber is still able to pause >> (suspend, yield, …) at that point, because the suspendable ports work in >> such a way, that no harm is done in doing so. Later the I/O work can be >> resumed (woken up from suspension). This quality had to be built into >> Guile first, before fibers were able to take advantage of it. >> >> Is this correct? > Yes, suspendable ports were first implemented in guile-2.2. They are > used by 8-sync (https://www.gnu.org/software/8sync/), guile-a-sync2 > (https://github.com/ChrisVine/guile-a-sync2/wiki) and fibers, and > possibly by other libraries. > > The basic arrangement is that if a port's file descriptor is not ready, > then its continuation is saved, and is resumed by the scheduler when it > becomes ready. fibers use epoll rather than POSIX select or poll for > this. > >> But I do not understand, why this is not the case with normal OS >> threads. Maybe it can be done but is not convenient or difficult to get >> right, to work with suspendable ports, when not using the fibers library? >> >> And why is simple-format not "suspendable-port-safe"? (What does a >> procedure need to do, in order to be "suspendable-port-safe"?) > simple-format does not suspend as it is implemented in C in libguile and > its continuation is not available to scheme code. There is a > list of those of guile's i/o procedures which do (and which do not) > suspend here, in the second and third paragraphs, although it does not > mention format/simple-format: > https://github.com/ChrisVine/guile-a-sync2/wiki/await-ports > (That library has nothing to do with fibers, but as mentioned above it > happens to use suspendable ports for similar purposes): > > None of this is likely to be relevant to your use case. > > [snip] >> With the parallel forms, isn't it the case, that at each call of such a >> form, new OS threads are created? In this case it might be a good idea >> to create a fibers scheduler and reuse it, if that is possible, so that >> I do not need to create my own process pool kind of thingy. > guile's parallel forms are implemented using futures, which use an > internal thread pool according to this: > https://www.gnu.org/software/guile/docs/master/guile.html/Futures.html#Futures > > "Internally, a fixed-size pool of threads is used to evaluate futures, > such that offloading the evaluation of an expression to another thread > doesn’t incur thread creation costs. By default, the pool contains one > thread per available CPU core, minus one, to account for the main > thread." > > Chris ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Guile fibers return values 2020-01-06 19:42 ` Zelphir Kaltstahl @ 2020-01-06 21:14 ` Chris Vine 2020-01-06 21:47 ` John Cowan 0 siblings, 1 reply; 16+ messages in thread From: Chris Vine @ 2020-01-06 21:14 UTC (permalink / raw) To: Zelphir Kaltstahl; +Cc: Guile User On Mon, 6 Jan 2020 20:42:17 +0100 Zelphir Kaltstahl <zelphirkaltstahl@posteo.de> wrote: [snip] > In Racket the futures have some limitations, where one needs to use a > different number type to enable them in some cases to run in parallel – > wait, I am looking for the link … here: > https://docs.racket-lang.org/guide/parallelism.html – Is there any > similar restriction for futures in Guile? I don't know. I should try it and see. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Guile fibers return values 2020-01-06 21:14 ` Chris Vine @ 2020-01-06 21:47 ` John Cowan 2020-01-06 22:45 ` Zelphir Kaltstahl 0 siblings, 1 reply; 16+ messages in thread From: John Cowan @ 2020-01-06 21:47 UTC (permalink / raw) To: Chris Vine; +Cc: Guile User Conceptually, parallelism and concurrency are two different and partly independent things. Parallelism refers to physically simultaneous execution, as when you throw a ball into the air in each hand and catch it in the same hand. Each throw-catch cycle is a parallel process (using "process" in the broad sense of the term). Concurrency, on the other hand, is *logically* simultaneous execution, as when you juggle three balls in one or two hands. Now the throw-catch cycle of each ball from one hand to the other is a concurrent process, and it is also a parallel process if you use two hands. If you are using one hand, however, there is no parallelism in juggling. To make matters more confusing, "futures" in Racket are for parallelism, whereas in Guile they are for concurrency. Guile "parallel" and friends are executed on futures (which are executed on OS threads), but use at most as many futures as there are CPUs, so physically simultaneous execution is at least encouraged if not actually guaranteed. Racket parallelism only operates until one of the parallel processes blocks or needs to synchronize (which includes things like allocating memory): they are not implemented on top of Racket threads, which are for concurrency (and have nothing to do with OS threads). A Scheme promise can be viewed as a type of parallel process that doesn't actually provide parallelism (and in fact my parallel pre-SRFI is called "parallel promises" and treats ordinary promises as a degenerate case) or as a future that doesn't start to execute until you wait for it to finish (and my futures pre-SRFI also treats promises as a degenerate case). On Mon, Jan 6, 2020 at 4:15 PM Chris Vine <vine35792468@gmail.com> wrote: > On Mon, 6 Jan 2020 20:42:17 +0100 > Zelphir Kaltstahl <zelphirkaltstahl@posteo.de> wrote: > [snip] > > In Racket the futures have some limitations, where one needs to use a > > different number type to enable them in some cases to run in parallel – > > wait, I am looking for the link … here: > > https://docs.racket-lang.org/guide/parallelism.html – Is there any > > similar restriction for futures in Guile? > > I don't know. I should try it and see. > > ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Guile fibers return values 2020-01-06 21:47 ` John Cowan @ 2020-01-06 22:45 ` Zelphir Kaltstahl 2020-01-07 1:36 ` John Cowan 0 siblings, 1 reply; 16+ messages in thread From: Zelphir Kaltstahl @ 2020-01-06 22:45 UTC (permalink / raw) To: John Cowan, Chris Vine; +Cc: Guile User Hello John! Thanks for your reply! On 06.01.2020 22:47, John Cowan wrote: > Conceptually, parallelism and concurrency are two different and partly > independent things. Parallelism refers to physically simultaneous > execution, as when you throw a ball into the air in each hand and > catch it in the same hand. Each throw-catch cycle is a parallel > process (using "process" in the broad sense of the term). > Concurrency, on the other hand, is *logically* simultaneous execution, > as when you juggle three balls in one or two hands. Now the > throw-catch cycle of each ball from one hand to the other is a > concurrent process, and it is also a parallel process if you use two > hands. If you are using one hand, however, there is no parallelism in > juggling. Yes, that is pretty clear. Parallelism and concurrency are not the same, I know that. One could also say, that one can have concurrent execution on a single core, even multiple processes, but cannot have parallel execution on that single core. With concurrency, even on one core, one needs to look out for many things like concurrent updates or mutating state, as one, in the general case, does not know when one process will be running and when the other. > To make matters more confusing, "futures" in Racket are for > parallelism, whereas in Guile they are for concurrency. Guile > "parallel" and friends are executed on futures (which are executed on > OS threads), but use at most as many futures as there are CPUs, so > physically simultaneous execution is at least encouraged if not > actually guaranteed. I think that is a typo maybe? The futures guide says: "The (ice-9 futures) module provides futures, a construct for fine-grain parallelism." Otherwise that would indeed be very confusing. Or do you say this, because on a single core machine, there would be no parallelism and thus one cannot say, that Guile's futures will enable parallelism in general, but can say, that they in general enable concurrency? > Racket parallelism only operates until one of the parallel processes > blocks or needs to synchronize (which includes things like allocating > memory): they are not implemented on top of Racket threads, which are > for concurrency (and have nothing to do with OS threads). Yes, as far as I understand, Racket threads are so called "green threads" (like in Python). To use multiple cores in the general case, one needs to make use of Racket's "places" instead, which are additional Racket VMs running. > A Scheme promise can be viewed as a type of parallel process that > doesn't actually provide parallelism (and in fact my parallel pre-SRFI > is called "parallel promises" and treats ordinary promises as a > degenerate case) or as a future that doesn't start to execute until > you wait for it to finish (and my futures pre-SRFI also treats > promises as a degenerate case). I think of "promises" as something that enables asynchronous execution. Don't beat me for this: "Just like in JavaScript" basically :D I don't know, if that notion is wrong in the Scheme context though. So far I found the following approaches to do things in parallel in GNU Guile: 1. futures 2. parallel forms (built on futures, probably "just" convenience) 3. fibers library I have not considered to look for "promises" yet, as I did not think them to be a parallelism construct or concept. However, you are mentioning them. Does that mean, that I should look into them as well or is it rather a general explanation to get the concepts cleanly separated? It seems not like they could parallelize any algorithm. At least not, if they share the character of JavaScript promises. And hello Chris! I have just checked, whether futures run in parallel for the example of the Racket docs and they seem to run in parallel, although the CPU is not 100% busy, probably because of other factors, like allocations: --------8<--------8<-------- (use-modules (ice-9 futures) ;; SRFI 19 for time related procedures (srfi srfi-19)) ;; Just defining a timing macro here to conveniently measure elapsed time of ;; evaluating expressions. (define-syntax time (syntax-rules () [(time expr expr* ...) (begin (define start-time (current-time time-monotonic)) expr expr* ... (define end-time (current-time time-monotonic)) (let* ([diff (time-difference end-time start-time)] [elapsed-ns (+ (/ (time-nanosecond diff) 1e9) (time-second diff))]) (display (format #t "~fs~%" elapsed-ns))))])) (define (mandelbrot iterations x y n) (let ([ci (- (/ (* 2.0 y) n) 1.0)] [cr (- (/ (* 2.0 x) n) 1.5)]) (let loop ([i 0] [zr 0.0] [zi 0.0]) (if (> i iterations) i (let ([zrq (* zr zr)] [ziq (* zi zi)]) (cond [(> (+ zrq ziq) 4.0) i] [else (loop (+ i 1) (+ (- zrq ziq) cr) (+ (* 2.0 zr zi) ci))])))))) (time (let ([f (future (lambda () (mandelbrot 10000000 62 501 1000)))]) (list (mandelbrot 10000000 62 500 1000) (touch f)))) (time (mandelbrot 10000000 62 501 1000)) --------8<--------8<-------- On my machine both timed expressions run in approximately the same time, which I conclude from, that 2 cores were used and the work was done in parallel. Regards, Zelphir ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Guile fibers return values 2020-01-06 22:45 ` Zelphir Kaltstahl @ 2020-01-07 1:36 ` John Cowan 0 siblings, 0 replies; 16+ messages in thread From: John Cowan @ 2020-01-07 1:36 UTC (permalink / raw) To: Zelphir Kaltstahl; +Cc: Guile User On Mon, Jan 6, 2020 at 5:45 PM Zelphir Kaltstahl <zelphirkaltstahl@posteo.de> wrote: I think that is a typo maybe? The futures guide says: > > "The (ice-9 futures) module provides futures, a construct for fine-grain > parallelism." > I know, and I think that's an error (or rather a confusing point) in the docs. It should say "for fine-grained concurrency". > Or do you say this, > because on a single core machine, there would be no parallelism and thus > one cannot say, that Guile's futures will enable parallelism in general, > but can say, that they in general enable concurrency? > Just so. > I think of "promises" as something that enables asynchronous execution. > Scheme promises are described in R7RS 4.2.5. They are basically just memoized closures. When you create one it sits there until you force it and then the closure is invoked and you get a result. If you force it again, you just get the result without any invocation. So you can think of this as a degenerate Guile future / Racket thread that doesn't even start until it is touched (by forcing it), or as a degenerate Racket future / Guile parallel construct that tries to run in parallel but is immediately unable to do so, so it has to wait until it is touched (also by forcing it). But promises don't provide you with either concurrency or parallelism in themselves, so they are only useful to help with understanding. Don't beat me for this: "Just like in JavaScript" basically :D > JS promises, I think, are futures. https://en.wikipedia.org/wiki/Futures_and_promises shows just how confusing the terminology is. John Cowan http://vrici.lojban.org/~cowan cowan@ccil.org Fundamental thinking is ha-ard. Let's go ideology-shopping. --Philosopher Barbie ^ permalink raw reply [flat|nested] 16+ messages in thread
* Guile fibers return values @ 2020-01-04 22:49 Zelphir Kaltstahl 2020-01-05 2:42 ` John Cowan 2020-01-14 10:59 ` Amirouche Boubekki 0 siblings, 2 replies; 16+ messages in thread From: Zelphir Kaltstahl @ 2020-01-04 22:49 UTC (permalink / raw) To: guile-user Hello Guile users! I have questions regarding the usage of the fibers library. It seems, that I cannot find any way to get a computation result back from a fiber. I also cannot find anything about how to get a value back from a fiber, except for channels. The examples include one example using the procedure `make-channel`, to create one channel for a client to send to a server and one channel to use for the server to send messages to the client. Are channels the only way to get a computation result back from a fiber? Should I be creating channels, which the spawned fiber then can use on its own, to asynchronously give me a result? This is an aside of my actual project currently. I want to parallelize some algorithm and want to make use of fibers for that, but in order to do that, I must understand how to run multiple fibers and get their results back. Can you give me an example, where fibers are used to split up a computation heavy task so that it is sped up, because of running on multiple cores? Regards, Zelphir ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Guile fibers return values 2020-01-04 22:49 Zelphir Kaltstahl @ 2020-01-05 2:42 ` John Cowan 2020-01-05 12:46 ` Zelphir Kaltstahl 2020-01-14 10:59 ` Amirouche Boubekki 1 sibling, 1 reply; 16+ messages in thread From: John Cowan @ 2020-01-05 2:42 UTC (permalink / raw) To: Zelphir Kaltstahl; +Cc: guile-user On Sat, Jan 4, 2020 at 5:50 PM Zelphir Kaltstahl <zelphirkaltstahl@posteo.de> wrote: I have questions regarding the usage of the fibers library. It seems, > that I cannot find any way to get a computation result back from a > fiber. I also cannot find anything about how to get a value back from a > fiber, except for channels. FIbers are much more like coroutines than subroutines: they don't return data, they pass it on. You *can* communicate by mutating shared data, but it's asking for trouble because of synchronization issues. Stick to communicating using channels, that's what they are for. Of course if your fiber both sends and receives on channels to the same fiber, you risk deadlock if you are not careful to stay exactly in sync and avoid output buffering. Exactly these rules apply to shell pipelines, probably the most widespread form of concurrency in programming as well as the simplest and most reliable. John Cowan http://vrici.lojban.org/~cowan cowan@ccil.org Deshil Holles eamus. Deshil Holles eamus. Deshil Holles eamus. Send us, bright one, light one, Horhorn, quickening, and wombfruit. (3x) Hoopsa, boyaboy, hoopsa! Hoopsa, boyaboy, hoopsa! Hoopsa, boyaboy, hoopsa! --Joyce, Ulysses, "Oxen of the Sun" ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Guile fibers return values 2020-01-05 2:42 ` John Cowan @ 2020-01-05 12:46 ` Zelphir Kaltstahl 0 siblings, 0 replies; 16+ messages in thread From: Zelphir Kaltstahl @ 2020-01-05 12:46 UTC (permalink / raw) To: John Cowan; +Cc: guile-user Thank you for your answer! I think the example with channels, which I sent later goes in the direction your are describing : ) Yes, I am not planning on modifying shared state and doing the whole mutex stuff. Regards, Zelphir On 1/5/20 3:42 AM, John Cowan wrote: > > > On Sat, Jan 4, 2020 at 5:50 PM Zelphir Kaltstahl > <zelphirkaltstahl@posteo.de <mailto:zelphirkaltstahl@posteo.de>> wrote: > > I have questions regarding the usage of the fibers library. It seems, > that I cannot find any way to get a computation result back from a > fiber. I also cannot find anything about how to get a value back > from a > fiber, except for channels. > > > FIbers are much more like coroutines than subroutines: they don't > return data, they pass it on. You *can* communicate by mutating > shared data, but it's asking for trouble because of synchronization > issues. Stick to communicating using channels, that's what they are > for. Of course if your fiber both sends and receives on channels to > the same fiber, you risk deadlock if you are not careful to stay > exactly in sync and avoid output buffering. Exactly these rules > apply to shell pipelines, probably the most widespread form of > concurrency in programming as well as the simplest and most reliable. > > > > John Cowan http://vrici.lojban.org/~cowan > cowan@ccil.org <mailto:cowan@ccil.org> > Deshil Holles eamus. Deshil Holles eamus. Deshil Holles eamus. > Send us, bright one, light one, Horhorn, quickening, and wombfruit. (3x) > Hoopsa, boyaboy, hoopsa! Hoopsa, boyaboy, hoopsa! Hoopsa, boyaboy, > hoopsa! > --Joyce, Ulysses, "Oxen of the Sun" > ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Guile fibers return values 2020-01-04 22:49 Zelphir Kaltstahl 2020-01-05 2:42 ` John Cowan @ 2020-01-14 10:59 ` Amirouche Boubekki 2020-01-15 0:04 ` Zelphir Kaltstahl 1 sibling, 1 reply; 16+ messages in thread From: Amirouche Boubekki @ 2020-01-14 10:59 UTC (permalink / raw) To: Zelphir Kaltstahl; +Cc: Guile User Hello Zelphir! Le sam. 4 janv. 2020 à 22:49, Zelphir Kaltstahl <zelphirkaltstahl@posteo.de> a écrit : > > Hello Guile users! > > I have questions regarding the usage of the fibers library. It seems, > that I cannot find any way to get a computation result back from a > fiber. I also cannot find anything about how to get a value back from a > fiber, except for channels. The examples include one example using the > procedure `make-channel`, to create one channel for a client to send to > a server and one channel to use for the server to send messages to the > client. > > Are channels the only way to get a computation result back from a fiber? > > Should I be creating channels, which the spawned fiber then can use on > its own, to asynchronously give me a result? > > This is an aside of my actual project currently. I want to parallelize > some algorithm and want to make use of fibers for that, but in order to > do that, I must understand how to run multiple fibers and get their > results back. > > Can you give me an example, where fibers are used to split up a > computation heavy task so that it is sped up, because of running on > multiple cores? > For that last bit, I have done the following in babelia: https://github.com/amirouche/guile-babelia/blob/87ae25b56777ab6072759bbe80bb80851d0d9174/babelia/pool.scm#L89-L108 I am wondering why the existing parallel for do not work for you: https://www.gnu.org/software/guile/manual/html_node/Parallel-Forms.html > Regards, > > Zelphir > > -- Amirouche ~ https://hyper.dev ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Guile fibers return values 2020-01-14 10:59 ` Amirouche Boubekki @ 2020-01-15 0:04 ` Zelphir Kaltstahl 0 siblings, 0 replies; 16+ messages in thread From: Zelphir Kaltstahl @ 2020-01-15 0:04 UTC (permalink / raw) To: Amirouche Boubekki; +Cc: Guile User Hi Amirouche! I am currently looking into using parallel forms, but am experiencing some trouble with it (see other e-mail about possible bug in parallel forms). I would like to have more control over the maximum number of in parallel running threads, when spawning tasks recursively. Parallel forms do not offer to limit the maximum, except for the expensive parallel forms. And those probably also do not offer it for recursive calls (in a task start again multiple tasks to run in parallel, which is the case for building the tree). I need to look at your code in detail to understand, but from the description or comments it looks promising for my use case. Thanks, Zelphir On 1/14/20 11:59 AM, Amirouche Boubekki wrote: > Hello Zelphir! > > Le sam. 4 janv. 2020 à 22:49, Zelphir Kaltstahl > <zelphirkaltstahl@posteo.de> a écrit : >> Hello Guile users! >> >> I have questions regarding the usage of the fibers library. It seems, >> that I cannot find any way to get a computation result back from a >> fiber. I also cannot find anything about how to get a value back from a >> fiber, except for channels. The examples include one example using the >> procedure `make-channel`, to create one channel for a client to send to >> a server and one channel to use for the server to send messages to the >> client. >> >> Are channels the only way to get a computation result back from a fiber? >> >> Should I be creating channels, which the spawned fiber then can use on >> its own, to asynchronously give me a result? >> >> This is an aside of my actual project currently. I want to parallelize >> some algorithm and want to make use of fibers for that, but in order to >> do that, I must understand how to run multiple fibers and get their >> results back. >> >> Can you give me an example, where fibers are used to split up a >> computation heavy task so that it is sped up, because of running on >> multiple cores? >> > For that last bit, I have done the following in babelia: > > https://github.com/amirouche/guile-babelia/blob/87ae25b56777ab6072759bbe80bb80851d0d9174/babelia/pool.scm#L89-L108 > > I am wondering why the existing parallel for do not work for you: > > https://www.gnu.org/software/guile/manual/html_node/Parallel-Forms.html > >> Regards, >> >> Zelphir >> >> > ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2020-01-15 0:04 UTC | newest] Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-01-05 1:30 Guile fibers return values Zelphir Kaltstahl 2020-01-05 12:33 ` Chris Vine 2020-01-05 12:58 ` Zelphir Kaltstahl 2020-01-05 14:28 ` Chris Vine [not found] ` <20200105142358.4ad96d15a23a0b947b2d55e3@gmail.com> 2020-01-05 18:22 ` Zelphir Kaltstahl 2020-01-05 21:45 ` Chris Vine 2020-01-06 19:42 ` Zelphir Kaltstahl 2020-01-06 21:14 ` Chris Vine 2020-01-06 21:47 ` John Cowan 2020-01-06 22:45 ` Zelphir Kaltstahl 2020-01-07 1:36 ` John Cowan -- strict thread matches above, loose matches on Subject: below -- 2020-01-04 22:49 Zelphir Kaltstahl 2020-01-05 2:42 ` John Cowan 2020-01-05 12:46 ` Zelphir Kaltstahl 2020-01-14 10:59 ` Amirouche Boubekki 2020-01-15 0:04 ` Zelphir Kaltstahl
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).