Elisp containers

Elisp containers

[Stefan Monnier]

If someone feels like they have too much time on their hands, I think
a great feature to develop would be Elisp containers.

This would be like running Elisp in a separate process, except that it's
not a separate process, so communication between two containers can be
very efficient (e.g. you can send a buffer from one container to the
other as efficiently as you can send an integer).

Potential applications:
- true concurrency
- safely running arbitrary Elisp code (e.g. bug#32495)
- discover what a package defines without actually defining those things
  (i.e. load the package into a container, then look at what got
  defined and throw away the container).


        Stefan

Re: Elisp containers

[John Wiegley]

>>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

SM> This would be like running Elisp in a separate process, except that it's
SM> not a separate process, so communication between two containers can be
SM> very efficient (e.g. you can send a buffer from one container to the other
SM> as efficiently as you can send an integer).

So the motivation is just efficiency? I think it costs <50ms to spawn an Emacs
child process using async.el to load and query something in it.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Elisp containers

[Stefan Monnier]

> SM> This would be like running Elisp in a separate process, except that it's
> SM> not a separate process, so communication between two containers can be
> SM> very efficient (e.g. you can send a buffer from one container to the other
> SM> as efficiently as you can send an integer).
> So the motivation is just efficiency?

Yes, I'm thinking of cases where there's fine-grained communication:
pass a buffer to the other container which does a quick change then
returns the buffer, etc...

> I think it costs <50ms to spawn an Emacs child process using async.el
> to load and query something in it.

Yes, maybe it's good enough, indeed.

Has anyone tried to use async for elisp--local-variables?
Same for Gnus and/or EWW?


        Stefan

Re: Elisp containers

[John Wiegley]

>>>>> "SM" == Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> I think it costs <50ms to spawn an Emacs child process using async.el to
>> load and query something in it.

SM> Yes, maybe it's good enough, indeed.

Conceivably, because that child process doesn't need to use windowing or any
other features, it would be possible to dump an even smaller Emacs binary
which only has support for "running headless".

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Elisp containers

[Eric Abrahamsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> If someone feels like they have too much time on their hands, I think
> a great feature to develop would be Elisp containers.
>
> This would be like running Elisp in a separate process, except that it's
> not a separate process, so communication between two containers can be
> very efficient (e.g. you can send a buffer from one container to the
> other as efficiently as you can send an integer).
>
> Potential applications:
> - true concurrency
> - safely running arbitrary Elisp code (e.g. bug#32495)
> - discover what a package defines without actually defining those things
>   (i.e. load the package into a container, then look at what got
>   defined and throw away the container).

I'm curious: could this be used to implement namespaces, as well?

Re: Elisp containers

[Tom Tromey]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

Stefan> If someone feels like they have too much time on their hands, I think
Stefan> a great feature to develop would be Elisp containers.

Stefan> This would be like running Elisp in a separate process, except that it's
Stefan> not a separate process, so communication between two containers can be
Stefan> very efficient (e.g. you can send a buffer from one container to the
Stefan> other as efficiently as you can send an integer).

I considered this when working on threads.  I thought I'd write to
discuss the problems I had with it; not to discourage anybody, but in
case somebody has thoughts on how to improve things.

I think the basic idea is to avoid data races while also avoiding the
overhead of a separate process.  So the design should keep things in
process (duh) and also reject shared mutable state; instead objects must
be explicitly sent between threads.  (If you lift this latter
requirement then you really just have what we have now.)


So, first, what environment should a new thread inherit?

If it is a clean environment -- say a copy of the dumped state -- then
that means that user changes won't affect threads.  And, threading code
will need to know exactly what things must be copied into the new
container.

This seemed unfortunate to me, not to mention hard on thread users,
because it's difficult in an open environment to know what may be
required.  (You can see echoes of this problem in the mhtml mode, where
it has to use a regexp on symbol names, eww, to capture locals.)

However, if it is not a clean environment, then thread creation will be
very expensive.  You will have to copy much of the heap into the new
thread.  (Perhaps the copying could be done lazily at the expense of
slowing down access to global variables.)


Second, sending a large object to another thread will also be slow.  The
buffer *itself* is not too bad, but buffers have properties and
buffer-local bindings, and these must all be deep copied.

One idea I had to reduce the cost here was to have a special case: when
a thread dies, let thread-join simply transfer the data from that thread
to its own heap.


Finally, this approach greatly reduces debuggability and the ability to
mess around.  Emacs would no longer be an open system -- threads would
have private data and there would be no way to access that.

Tom

Re: Elisp containers

[Vladimir Sedach]

> I think the basic idea is to avoid data races while also avoiding the
> overhead of a separate process.  So the design should keep things in
> process (duh) and also reject shared mutable state; instead objects must
> be explicitly sent between threads.  (If you lift this latter
> requirement then you really just have what we have now.)

This problem boils down to "what is the right way to virtualize a
single address space Lisp system?"

> So, first, what environment should a new thread inherit?
>
> If it is a clean environment -- say a copy of the dumped state -- then
> that means that user changes won't affect threads.  And, threading code
> will need to know exactly what things must be copied into the new
> container.
>
> This seemed unfortunate to me, not to mention hard on thread users,
> because it's difficult in an open environment to know what may be
> required.  (You can see echoes of this problem in the mhtml mode, where
> it has to use a regexp on symbol names, eww, to capture locals.)

> However, if it is not a clean environment, then thread creation will be
> very expensive.  You will have to copy much of the heap into the new
> thread.  (Perhaps the copying could be done lazily at the expense of
> slowing down access to global variables.)

One way to solve the problems of "which environment to start from?"
and expensive copying is to implement environments as a fully
persistent (AKA purely functional) table. This will let you do
roughly the equivalent of fork(2) in a single address space quickly
(with the real cost amortized over the whole system). You can also
use this for parent/child fork parallelism where the children inherit
the parent's dynamic bindings.

Having single address space share-nothing threads IMO defeats the
main benefits of a single address space, which are: eliminating data
duplication, and zero copy message passing. It is also a lot more
work to implement than using multiple Emacs OS processes.
Virtualizing the environment with a shared heap is an IMO interesting
idea that has not been well explored yet, but it requires a very
different approach to Lisp system functions (they will all have to be
based around immutable data structures), and more to the point is not
relevant to "Elisp containers."

> One idea I had to reduce the cost here was to have a special case: when
> a thread dies, let thread-join simply transfer the data from that thread
> to its own heap.

Cilk has a very well thought out model of lightweight thread
forking/joining using cactus stacks (an idea derived from Interlisp's
spaghetti stack), where child threads return their data to the parent.

> Finally, this approach greatly reduces debuggability and the ability to
> mess around.  Emacs would no longer be an open system -- threads would
> have private data and there would be no way to access that.

The debugger could point to the environment of the thread from which
it was invoked. The same could be done for REPLs. So the whole
tool set would have to be changed to accommodate this virtualization.

Multiple Emacs processes have the same problem, but there are already
working solutions that can be adapted for async.el. Something like
the SWANK protocol can be used to provide debugging and a REPL for
multiple Emacs processes, and not just on the same machine but
remotely as well. The nice thing is that the Emacs processes already
show up in the process list, and would get their own names in SWANK,
which would make managing them as easy as managing containers.

Vladimir

Re: Elisp containers

[Tom Tromey]

>>>>> "Vladimir" == Vladimir Sedach <vas@oneofus.la> writes:

Vladimir> One way to solve the problems of "which environment to start from?"
Vladimir> and expensive copying is to implement environments as a fully
Vladimir> persistent (AKA purely functional) table. This will let you do
Vladimir> roughly the equivalent of fork(2) in a single address space quickly
Vladimir> (with the real cost amortized over the whole system).

One other consideration in this case is that one must apply the changes
to the actually existing Emacs.  Reading the portable dumper thread or
the variable binding code makes this seem less possible.  Though perhaps
I am pessimistic.

Vladimir> Virtualizing the environment with a shared heap is an IMO interesting
Vladimir> idea that has not been well explored yet, but it requires a very
Vladimir> different approach to Lisp system functions (they will all have to be
Vladimir> based around immutable data structures)

I should have mentioned immutable data structures.  This seems like a
difficult change as well.

In practice I guess I would not worry too much about existing elisp
code.  Probably if one is reaching for threads or containers, one is
willing to adapt one's code to immutability as well.

But I wonder how it would be implemented.  Tag bits are in short supply
and there isn't leftover space in a cons for a flag AFAIK.  Other types
might not be as troublesome.

>> Finally, this approach greatly reduces debuggability and the ability to
>> mess around.  Emacs would no longer be an open system -- threads would
>> have private data and there would be no way to access that.

Vladimir> The debugger could point to the environment of the thread from which
Vladimir> it was invoked. The same could be done for REPLs. So the whole
Vladimir> tool set would have to be changed to accommodate this virtualization.

Thanks, this is a nice idea.

Tom

Re: Elisp containers

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Potential applications:
  > - true concurrency
  > - safely running arbitrary Elisp code (e.g. bug#32495)
  > - discover what a package defines without actually defining those things
  >   (i.e. load the package into a container, then look at what got
  >   defined and throw away the container).

Also, compilation of files that define macros, without altering the
overall Lisp environment.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Elisp containers

[Helmut Eller]

On Fri, Sep 07 2018, Stefan Monnier wrote:

> If someone feels like they have too much time on their hands, I think
> a great feature to develop would be Elisp containers.
>
> This would be like running Elisp in a separate process, except that it's
> not a separate process, so communication between two containers can be
> very efficient (e.g. you can send a buffer from one container to the
> other as efficiently as you can send an integer).

Would containers be isolated from each other like (unix) processes or
would they share something, like immutable data?  Which parts would be
shared?

Helmut

Re: Elisp containers

[Stefan Monnier]

> Would containers be isolated from each other like (unix) processes or
> would they share something, like immutable data?  Which parts would be
> shared?

It's all hypothetical, so it depends on which possible world
you choose.  But ideally, they should share all immutable data and be
able to transfer mutable data by only changing its ownership.


        Stefan