Asynchronous insertion / saving of buffers

Asynchronous insertion / saving of buffers

[Michael Albinus]

Hi,

these days I'm working on Bug#9617, which asks for non-blocking
(asynchronous) insertion or saving of buffers. This makes sense for
large remote files, because Emacs would be blocked for a while during
these operations.

The implementation for `insert-file-contents' works pretty well
meanwhile (technically, an asynchronous process copies the remote file
into a local temporary file in the background, and a corresponding
process sentinel inserts this local file into the buffer once the
process has finished).

However, this is not sufficient. File name handlers, like Tramp, know
only primitive operations. In the environment where these primitive
operations are called, further actions will be applied, assuming that
the primitive operation has finished once it returns. In case of
`insert-file-contents', further operations will be applied, which fail
in the asynchronous case due to the missing buffer contents. Examples:

- Call of `after-find-file', which expands for example tar files

- Operations from vc-*, which check a status (like reading and parsing
  dirstate in vc-bzr)

Obviously, we need some design how to handle asynchronous file
operations. Any proposals?

Best regards, Michael.

Re: Asynchronous insertion / saving of buffers

[Stefan Monnier]

> Obviously, we need some design how to handle asynchronous file
> operations. Any proposals?

I think it requires fairly major changes: you can't just make
insert-file-contents asynchronous because its behavior is defined (and
assumed by callers) to be synchronous.

I.e. you need a new way to "insert asynchronously".  It can be via a new
function `insert-file-asynchronously' or via a new argument (or special
arg value) of insert-file-contents, so that insert-file-contents knows
that the caller expects an async behavior.

And then all the callers that want to support async operations need to
be rewritten (and some of their callers as well).

Maybe working on the concurrency branch would be easier and more useful.


        Stefan

Re: Asynchronous insertion / saving of buffers

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Obviously, we need some design how to handle asynchronous file
>> operations. Any proposals?
>
> I think it requires fairly major changes: you can't just make
> insert-file-contents asynchronous because its behavior is defined (and
> assumed by callers) to be synchronous.
>
> I.e. you need a new way to "insert asynchronously".  It can be via a new
> function `insert-file-asynchronously' or via a new argument (or special
> arg value) of insert-file-contents, so that insert-file-contents knows
> that the caller expects an async behavior.
>
> And then all the callers that want to support async operations need to
> be rewritten (and some of their callers as well).

All of this agreed. That's why I've started with my question.

> Maybe working on the concurrency branch would be easier and more useful.

Yes, this might help. But you still need a kind of infrastructure as you
have pointed out above. Would those new APIs differ, when the
implementation is based on threads, instead of of what I'm doing now?

>         Stefan

Best regards, Michael.

Re: Asynchronous insertion / saving of buffers

[Stefan Monnier]

> Yes, this might help. But you still need a kind of infrastructure as you
> have pointed out above.  Would those new APIs differ, when the
> implementation is based on threads, instead of of what I'm doing now?

With threads you could "just" do something like

 (run-thread (lambda ()
               (insert-file-contents ...)
               ...))

so there'd be no need to provide new asynchronous versions of things
like insert-file-contents.


        Stefan

Re: Asynchronous insertion / saving of buffers

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> With threads you could "just" do something like
>
>  (run-thread (lambda ()
>                (insert-file-contents ...)
>                ...))
>
> so there'd be no need to provide new asynchronous versions of things
> like insert-file-contents.

But you still need a mean to inform the caller, that the parallel action
has been finished. And you need a mean to lock the buffer, while
insert-file-contents modifies it.

I haven't read the doc in the concurrency branch, maybe this would
answer my questions.

>         Stefan

Best regards, Michael.

Re: Asynchronous insertion / saving of buffers

[Stefan Monnier]

>> With threads you could "just" do something like
>> (run-thread (lambda ()
>> (insert-file-contents ...)
>> ...))
>> so there'd be no need to provide new asynchronous versions of things
>> like insert-file-contents.
> But you still need a mean to inform the caller, that the parallel action
> has been finished. And you need a mean to lock the buffer, while
> insert-file-contents modifies it.

Of course, but none of that is specific to insert-file-contents.
It's just part of the functionality that is generally needed for concurrency.

> I haven't read the doc in the concurrency branch, maybe this would
> answer my questions.

I don't think we'd have any locking, instead you'd have to be careful
not to touch the buffer until insert-file-contents is over.
But you could add a `busy' buffer-local variable and use it as a lock if
you feel like it.

Also to inform other code that insert-file-contents is done, you can
simply use a `done' variable which you set to t when it's done, so the
other code can check when it's over.  But a more likely way to work is
to put the "things to do when the insert-file-contents is done" in the
second "..." of my sample code.


        Stefan

Re: Asynchronous insertion / saving of buffers

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I don't think we'd have any locking, instead you'd have to be careful
> not to touch the buffer until insert-file-contents is over.
> But you could add a `busy' buffer-local variable and use it as a lock if
> you feel like it.

Likely, the buffer shall be read-only while the other thread runs. And
maybe we shall add a more instructive warning when somebody tries to
modify the buffer that time. All this could be wrapped by something
like insert-file-contents-asynchronically.

> Also to inform other code that insert-file-contents is done, you can
> simply use a `done' variable which you set to t when it's done, so the
> other code can check when it's over.  But a more likely way to work is
> to put the "things to do when the insert-file-contents is done" in the
> second "..." of my sample code.

I see.

Well, you're right, this makes more sense than extending a file name
handler for insert-file-contents. Thanks.

>         Stefan

Best regards, Michael.

Re: Asynchronous insertion / saving of buffers

[Ted Zlatanov]

On Fri, 13 Apr 2012 14:44:31 +0200 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> However, this is not sufficient. File name handlers, like Tramp, know
MA> only primitive operations. In the environment where these primitive
MA> operations are called, further actions will be applied, assuming that
MA> the primitive operation has finished once it returns. In case of
MA> `insert-file-contents', further operations will be applied, which fail
MA> in the asynchronous case due to the missing buffer contents. Examples:

MA> - Call of `after-find-file', which expands for example tar files

MA> - Operations from vc-*, which check a status (like reading and parsing
MA>   dirstate in vc-bzr)

MA> Obviously, we need some design how to handle asynchronous file
MA> operations. Any proposals?

Maybe you can approach this similarly to url-future.el?

I intended to make it generic but only had use for the URL retrieval
case; you'll see it's easy to adapt it to be generic.  Futures map well
to combinations of synchronous/asynchronous operations, in my
experience, and they can be chained.

Ted

Re: Asynchronous insertion / saving of buffers

[Michael Albinus]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Maybe you can approach this similarly to url-future.el?
>
> I intended to make it generic but only had use for the URL retrieval
> case; you'll see it's easy to adapt it to be generic.  Futures map well
> to combinations of synchronous/asynchronous operations, in my
> experience, and they can be chained.

Maybe. Unfortunately, I don't understand how to use it, reading the
source. Do you have an example which demonstrates its usage?

> Ted

Best regards, Michael.

Re: Asynchronous insertion / saving of buffers

[Ted Zlatanov]

On Fri, 20 Apr 2012 15:49:11 +0200 Michael Albinus <michael.albinus@gmx.de> wrote: 

MA> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Maybe you can approach this similarly to url-future.el?
>> 
>> I intended to make it generic but only had use for the URL retrieval
>> case; you'll see it's easy to adapt it to be generic.  Futures map well
>> to combinations of synchronous/asynchronous operations, in my
>> experience, and they can be chained.

MA> Maybe. Unfortunately, I don't understand how to use it, reading the
MA> source. Do you have an example which demonstrates its usage?

Sorry, I have done a bad job documenting it :)  If the explanation below
is helpful, I'll change the docs in the file.

The idea is not new; Scheme has `delay' and `force' to do similar work.
It's a way to promise work but not necessarily use it immediately.

So let's make a future to promise the evaluation of 2+2:

#+begin_src lisp
(let ((f (make-url-future :value (lambda () (+ 2 2))
                          :callback (lambda (future) (debug future))
                          :errorback (lambda (future &rest error)
                                       (debug future error)))))

  (message "Future completion status before call: %s, value %s"
           (url-future-completed-p f)
           (url-future-value f))
  (url-future-call f)
  (message "Future completion status after call: %s, value %s"
           (url-future-completed-p f)
           (url-future-value f)))

#+end_src

This will print (with a debugger call you can `c' out of, in the callback):

Future completion status before call: nil, value (lambda nil (+ 2 2))
Future completion status after call: t, value 4

Replace (+ 2 2) with (+ 2 nil) in the value to see error handling.

Emacs Lisp doesn't have support for delayed evaluation, so this was a
way to work around that limitation and provide a generic interface.  I
don't know if the language could support tighter integration with
delayed evaluations without core changes.

Futures can be cancelled in this implementation.

It's under url-future-* because we wanted to see how useful it would
prove before making it more prominent.  If you find this approach
useful, let me know; if not and you think it could be useful with some
API or code changes then that's good to know as well.

Thanks
Ted