Re: continuation passing in Emacs vs. JUST-THIS-ONE

[Tomas Hlavaty <tom@logand.com>]

On Mon 10 Apr 2023 at 22:53, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>> And the use `await` above means that your Emacs will block while waiting
>>> for one result.  `futur-let*` instead lets you compose async operations
>>> without blocking Emacs, and thus works more like Javascript's `await`.
>> Blocking the current thread for one result is fine, because all the
>> futures already run in other threads in "background" so there is nothing
>> else to do.
>
> You can't know that.  There can be other async processes whose
> filters should be run, timers to be executed, other threads to run,
> ...

I do know that, because in this case I have been talking about the
specific case where the future runs in a thread, i.e. my async-thread
and await-thread code.  Looks like similar use-case partially sketched
in futur-wait, unfortunately without working code to try.

>> If you mean that you want to use the editor at the same time, just run
>> the example in another thread.
>
> The idea is to use `futur.el` *instead of* threads.

What do you mean?
I can see thread-join and thread-signal in futur.el.

It is useful to acknowledge, that there are 3 different use-cases:
a) asynchronous processes
b) threads
c) iter

My impression was that futur.el was trying to address a) and b) but now
you say it does address a) only.  That is rather limited.

>> No, the iter case does map directly to futures:
>>
>> (await
>>  (async-iter
>>    (let ((a (async-iter
>>               (message "a1")
>>               (await-iter (sleep-iter3 3))
>>               (message "a2")
>>               1))
>>          (b (async-iter
>>               (message "b1")
>>               (let ((c (async-iter
>>                          (message "c1")
>>                          (await-iter (sleep-iter3 3))
>>                          (message "c2")
>>                          2)))
>>                 (message "b2")
>>                 (+ 3 (await-iter c))))))
>>      (+ (await-iter a) (await-iter b)))))
>
> I must say I don't understand this example: in which sense is it using
> "iter"?  I don't see any `iter-yield`.

await-iter and async-iter macros are using iter under the hood.

One could use iter-yield explicitly too but that example
does not need to do that.

I have already sent the definitions of those macros earlier but here the
whole thing self-contained:

;;; -*- lexical-binding: t -*-

(defmacro async-iter (&rest body)
  (declare (indent 0))
  `(let ((i (iter-make (iter-yield (progn ,@body))))
         (z 'EAGAIN))
     (setq z (iter-next i))
     (lambda ()
       (when (eq 'EAGAIN z)
         (setq z (iter-next i)))
       (unless (eq 'EAGAIN z)
         (iter-close i))
       z)))

(defmacro await-iter (future)
  (let ((f (gensym)))
    `(let ((,f ,future))
       (let (z)
         (while (eq 'EAGAIN (setq z (funcall ,f)))
           (iter-yield 'EAGAIN))
         z))))

(defun sleep-iter3 (sec)
  (async-iter
    (let ((end (+ (float-time (current-time)) sec)))
      (while (< (float-time (current-time)) end)
        (iter-yield 'EAGAIN)))
    (iter-yield sec)))

(defun await-in-background (future &optional callback secs repeat)
  (let ((z (funcall future)))
    (if (eq 'EAGAIN z)
        (let (timer)
          (setq timer (run-with-timer
                       (or secs 0.2)
                       (or repeat 0.2)
                       (lambda ()
                         (let ((z (funcall future)))
                           (unless (eq 'EAGAIN z)
                             (cancel-timer timer)
                             (when callback
                               (funcall callback z))))))))
      (when callback
        (funcall callback z)))
    z))

(await-in-background
 (async-iter
   (let ((a (async-iter
              (message "a1")
              (await-iter (sleep-iter3 3))
              (message "a2")
              1))
         (b (async-iter
              (message "b1")
              (let ((c (async-iter
                         (message "c1")
                         (await-iter (sleep-iter3 3))
                         (message "c2")
                         2)))
                (message "b2")
                (+ 3 (await-iter c))))))
     (message "Result = %s" (+ (await-iter a) (await-iter b))))))

> `futur.el` also "queues the continuations in the event loop".

I get:

futur.el:97:8: Warning: the function ‘funcall-later’ is not known to be
    defined.

>>>> Calling await immediately after async is useless (simply use blocking
>>>> call).  The point of future is to make the distance between those calls
>>>> as big as possible so that the sum of times in the sequential case is
>>>> replaced with max of times in the parallel case.
>>> You're looking for parallelism.  I'm not.
>> What do you mean exactly?
>
> That `futur.el` is not primarily concerned with allowing you to run
> several subprocesses to exploit your multiple cores.  It's instead
> primarily concerned with making it easier to write asynchronous code.
>
> One of the intended use case would be for completion tables to return
> futures (which, in many cases, will have already been computed
> synchronously, but not always).
>
>> I am asking because:
>>
>> https://wiki.haskell.org/Parallelism_vs._Concurrency
>>
>>    Warning: Not all programmers agree on the meaning of the terms
>>    'parallelism' and 'concurrency'. They may define them in different
>>    ways or do not distinguish them at all.
>
> Yet I have never heard of anyone disagree with the definitions given at
> the beginning of that very same page.  More specifically those who may
> disagree are those who didn't know there was a distinction :-)

strawman

I was asking in order to understand why you dismissed my examples by
saying:

   You're looking for parallelism.  I'm not.

It looks to me that the reason is that futur.el cannot do those things
demonstrated in my examples.

>> But it seems that you insist on composing promises sequentially:
>
> No, I'm merely making it easy to do that.
>
>> Also futur.el does seems to run callbacks synchronously:
>
> I don't think so: it runs them via `funcall-later`.
>
>> In this javascript example, a and b appear to run in parallel (shall I
>> say concurrently?):
>>
>> function sleep(sec) {
>>   return new Promise(resolve => {
>>     setTimeout(() => {resolve(sec);}, sec * 1000);
>>   });
>> }
>> async function test() {
>>   const a = sleep(9);
>>   const b = sleep(8);
>>   const z = await a + await b;
>>   console.log(z);
>> }
>> test();
>>
>> Here the console log will show 17 after 9sec.
>> It will not show 17 after 17sec.
>>
>> Can futur.el do that?
>
> Of course.  You could do something like
>
>       (futur-let*
>           ((a (futur-let* ((_ <- (futur-process-make
>                                  :command '("sleep" "9"))))
>                  9))
>            (b (futur-let* ((_ <- (futur-process-make
>                                  :command '("sleep" "8"))))
>                  8))
>            (a-val <- a)
>            (b-val <- b))
>         (message "Result = %s" (+ a-val b-val))))

So will futur.el take 9sec or 17sec?
I cannot test this because it does not work:

I get:

Debugger entered--Lisp error: (wrong-type-argument stringp nil)
  #<subr make-process>(:sentinel #f(compiled-function (proc state) #<bytecode -0x17e00415238e9184>) :command ("sleep" "9"))
  make-process--with-editor-process-filter(#<subr make-process> :sentinel #f(compiled-function (proc state) #<bytecode -0x17e00415238e9184>) :command ("sleep" "9"))
  apply(make-process--with-editor-process-filter #<subr make-process> (:sentinel #f(compiled-function (proc state) #<bytecode -0x17e00415238e9184>) :command ("sleep" "9")))
  make-process(:sentinel #f(compiled-function (proc state) #<bytecode -0x17e00415238e9184>) :command ("sleep" "9"))
  apply(make-process :sentinel #f(compiled-function (proc state) #<bytecode -0x17e00415238e9184>) (:command ("sleep" "9")))
  #f(compiled-function (f) #<bytecode 0x5ccc7625eec9b18>)(#s(futur :clients nil :value nil))
  futur-new(#f(compiled-function (f) #<bytecode 0x5ccc7625eec9b18>))
  futur-process-make(:command ("sleep" "9"))
  (futur-bind (futur-process-make :command '("sleep" "9")) #'(lambda (_) 9))
  (let ((a (futur-bind (futur-process-make :command '("sleep" "9")) #'(lambda (_) 9)))) (let ((b (futur-bind (futur-process-make :command '("sleep" "8")) #'(lambda (_) 8)))) (futur-bind a #'(lambda (a-val) (futur-bind b #'(lambda ... ...))))))
  (progn (let ((a (futur-bind (futur-process-make :command '("sleep" "9")) #'(lambda (_) 9)))) (let ((b (futur-bind (futur-process-make :command '...) #'(lambda ... 8)))) (futur-bind a #'(lambda (a-val) (futur-bind b #'...))))))
  (setq elisp--eval-defun-result (progn (let ((a (futur-bind (futur-process-make :command '...) #'(lambda ... 9)))) (let ((b (futur-bind (futur-process-make :command ...) #'...))) (futur-bind a #'(lambda (a-val) (futur-bind b ...)))))))
  elisp--eval-defun()
  eval-defun(nil)
  funcall-interactively(eval-defun nil)
  command-execute(eval-defun)

>> I think that your confusion is caused by the decision that
>> futur-process-make yields exit code.  That is wrong, exit code is
>> logically not the resolved value (promise resolution), it indicates
>> failure (promise rejection).
>
> Not necessarily, it all depends on what the process is doing.
>
> Similarly the "intended return value" of a process will depend on what
> the process does.  In some cases it will be the stdout, but I see no
> reason to restrict my fundamental function to such a choice.

This overgeneralized thinking is beyond usefulness and harmfully leads
to the problem of how to maintain state.  It is better to have the
process future return the output on success and error on failure.  The
error can contain the error code which covers the specialized use-case
you were after.

> It's easy to build on top of `futur-process-make` a higher-level
> function which returns the stdout as the result of the future.

It might be easy but unnecessary.  That overgeneralized thinking also
lead you astray:

   From: Stefan Monnier <monnier@iro.umontreal.ca>
   Date: Thu, 16 Mar 2023 23:08:25 -0400
   id:jwvpm98nlqz.fsf-monnier+emacs@gnu.org

   BTW the above code can't work right now.  Part of the issue is the
   management of `current-buffer`: should the composition of futures
   with `futur-let*` save&restore `current-buffer` to mimic more closely
   the behavior one would get with plain old sequential execution?  If
   so, should we do the same with `point`?  What about other such state?

It is better not to open this can of worms.