obby

obby

[Richard M Stallman]

Gobby is a collaborative editing system which
uses a library called obby.  (See gobby.0x539.de/ .)

Is it feasible to make this work in Emacs?

Re: obby

[Jeff Kowalczyk]

On Sun, 10 May 2009 15:53:21 -0400, Richard M Stallman wrote:
> Gobby is a collaborative editing system which
> uses a library called obby.  (See gobby.0x539.de/ .)
> 
> Is it feasible to make this work in Emacs?

Phil Hagelberg previously released an Emacs implementation compatible with
obby-0.3:

 Re: [obby-users] Emacs implementation
 Phil Hagelberg
 Thu, 04 May 2006 17:16:11 -0700

The current URL is:

 http://dev.technomancy.us/ebby

Re: obby

[Jeff Kowalczyk]

It seems that obby development for Gobby has been superceded by Infinote,
a protocol and provided standalone server implementation:

  http://gobby.0x539.de/trac/wiki/Infinote

  Infinote is the name for the protocol replacing obby in Gobby 0.5.0. It
  has been redesigned from scratch and supports several features that obby
  did not, such as Undo or a hierarchical document tree.

Re: obby

[Phil Hagelberg]

Jeff Kowalczyk <jtk@yahoo.com> writes:

> On Sun, 10 May 2009 15:53:21 -0400, Richard M Stallman wrote:
>> Gobby is a collaborative editing system which
>> uses a library called obby.  (See gobby.0x539.de/ .)
>> 
>> Is it feasible to make this work in Emacs?
>
> Phil Hagelberg previously released an Emacs implementation compatible with
> obby-0.3:

My implementation has since moved:

http://github.com/technomancy/dotfiles/blob/63aac563d5f2417a7500942d2ac392c118d1d539/.emacs.old/ebby.el

There are two major problems with my implementation. First, it's only
compatible with a version of the obby protocol that is over three years
out of date. Secondly, my knowledge of the protocol came from
packet-sniffing since the protocol was undocumented at the time and I
don't know C, so it doesn't do any inclusion transformation. This is
necessary to support handling conflicts when two people edit the same
portion of the document at the same time. This means that each client
connected will eventually converge from having the same text for any
given session unless they are on a near-zero-latency network.

I had planned to revisit the problem once the new implementation of the
obby protocol and client (now called Infinote) had been released. But
implementing inclusion transformation is very difficult.

By the way, this is a textbook case for why we would want to support
dynamic linking. The reference implementation of the Infinote protocol,
libinfinote, is released under the LGPL, so it would save a great deal
of effort to be able to use that rather than creating a new,
independent, under-manned implementation from scratch. It's especially
hard to get motivated to sink that much effort into an alternate
implementation when you know it would be much easier and more robust to
link to an existing Free one.

-Phil

Re: obby

[Stefan Monnier]

> By the way, this is a textbook case for why we would want to support
> dynamic linking. The reference implementation of the Infinote protocol,
> libinfinote, is released under the LGPL, so it would save a great deal
> of effort to be able to use that rather than creating a new,
> independent, under-manned implementation from scratch. It's especially
> hard to get motivated to sink that much effort into an alternate
> implementation when you know it would be much easier and more robust to
> link to an existing Free one.

Short of dynamic-linking there are 2 options:
- link at compile-time, as we do for png, svg, ...
- write a wrapper to turn the C API into a little executable that speaks
  "the same API" over its stdin/stdout, so you can use it via
  start-process.


        Stefan "who hopes we will find some way to allow dynamic loading
                of libraries at some point, maybe with a similar
                approach as what is being designed for javascript"

Re: obby

[joakim]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> By the way, this is a textbook case for why we would want to support
>> dynamic linking. The reference implementation of the Infinote protocol,
>> libinfinote, is released under the LGPL, so it would save a great deal
>> of effort to be able to use that rather than creating a new,
>> independent, under-manned implementation from scratch. It's especially
>> hard to get motivated to sink that much effort into an alternate
>> implementation when you know it would be much easier and more robust to
>> link to an existing Free one.
>
> Short of dynamic-linking there are 2 options:
> - link at compile-time, as we do for png, svg, ...
> - write a wrapper to turn the C API into a little executable that speaks
>   "the same API" over its stdin/stdout, so you can use it via
>   start-process.

I'm interested in this feature, and I could help write the C wrapper.

>
>
>         Stefan "who hopes we will find some way to allow dynamic loading
>                 of libraries at some point, maybe with a similar
>                 approach as what is being designed for javascript"
>

Do you have any links explaining this aproach?

Another aproach I was thinking of was writing some kind of plugin for
the Swig wrapper generator for elisp. The idea is that Swig would be
persuaded to generate C stubs and low-level elisp api:s from a C library
.h file, like Swig already does for Java, Python and so on. I think this
aproach would both be fairly convenient(not as convenient as dynamic
linking of course) and safe from potential threats from dynamic linking.

-- 
Joakim Verona

Re: obby

[Richard M Stallman]

    Phil Hagelberg previously released an Emacs implementation compatible with
    obby-0.3:

     Re: [obby-users] Emacs implementation
     Phil Hagelberg
     Thu, 04 May 2006 17:16:11 -0700

     http://dev.technomancy.us/ebby

That is good news.  Would someone here like to try it and report?

Re: obby

[Richard M Stallman]

    Another aproach I was thinking of was writing some kind of plugin for
    the Swig wrapper generator for elisp. The idea is that Swig would be
    persuaded to generate C stubs and low-level elisp api:s from a C library
    .h file, like Swig already does for Java, Python and so on. I think this
    aproach would both be fairly convenient(not as convenient as dynamic
    linking of course) and safe from potential threats from dynamic linking.

I can see how generating wrappers could be useful, but how could it help
deal with the potential threat of non-free add-ons?

Re: obby

[joakim]

Richard M Stallman <rms@gnu.org> writes:

>     Another aproach I was thinking of was writing some kind of plugin for
>     the Swig wrapper generator for elisp. The idea is that Swig would be
>     persuaded to generate C stubs and low-level elisp api:s from a C library
>     .h file, like Swig already does for Java, Python and so on. I think this
>     aproach would both be fairly convenient(not as convenient as dynamic
>     linking of course) and safe from potential threats from dynamic linking.
>
> I can see how generating wrappers could be useful, but how could it help
> deal with the potential threat of non-free add-ons?

The wrappers would be statically linked. This would be the same
situation we have today, that is, elisp on top of statically linked
wrappers for dynamic libraries, for instance librsvg.

I should note that I dont fully understand why the GPL cant be relied on
to prevent linking to non-free dynamic libraries, so my analysis might
be flawed.

>
-- 
Joakim Verona

Re: obby

[Richard M Stallman]

    I should note that I dont fully understand why the GPL cant be relied on
    to prevent linking to non-free dynamic libraries, so my analysis might
    be flawed.

It is just that the legal argument is not a slam dunk in this case.

Re: obby

[Richard M Stallman]

    I had planned to revisit the problem once the new implementation of the
    obby protocol and client (now called Infinote) had been released. But
    implementing inclusion transformation is very difficult.

Can you tell us more about this problem?

    By the way, this is a textbook case for why we would want to support
    dynamic linking. The reference implementation of the Infinote protocol,
    libinfinote, is released under the LGPL, so it would save a great deal
    of effort to be able to use that rather than creating a new,
    independent, under-manned implementation from scratch.

We could link with it non-dynamically too.

I don't understand the structure of the situation.  Did you
reimplement the same thing in Lisp?

Re: obby

[Christian Lynbech]

>>>>> Richard M Stallman <rms@gnu.org> writes:

>     I had planned to revisit the problem once the new implementation of the
>     obby protocol and client (now called Infinote) had been released.

What is the main target: to specifically make a gobby compatible client
or to more generally just add collaborative editing to emacs?


------------------------+-----------------------------------------------------
Christian Lynbech       | christian #\@ defun #\. dk
------------------------+-----------------------------------------------------
Hit the philistines three times over the head with the Elisp reference manual.
                                        - petonic@hal.com (Michael A. Petonic)

Re: obby

[Phil Hagelberg]

Richard M Stallman <rms@gnu.org> writes:

>     I had planned to revisit the problem once the new implementation of the
>     obby protocol and client (now called Infinote) had been released. But
>     implementing inclusion transformation is very difficult.
>
> Can you tell us more about this problem?

The first part involves keeping a lot of state. Every edit that could be
re-applied (by someone else sending a conflicting edit to the same
portion of the document before yours reached the server) would need to
be stored along with some metadata. (It may be possible to tie into
Emacs' existing undo mechanism for this part if you were able to add
some metadata to each action.)

But then if you are notified of a conflict, you need to un-apply your
changes, apply the canonical ones you received from the server, and
transform your edits so they apply cleanly. This is pretty
complicated. It's analogous to a rebase operation if you're familiar
with the concept from some distributed version control systems, only it
applies on the document level rather than across a whole
repository. There are more edge cases that result when re-applied edits
need to be split to ensure consistency; I am not sure I understand these
fully.

Details about the transformation algorithm are available on the obby wiki:

  http://gobby.0x539.de/trac/wiki/AnnotatedObbySession

>     By the way, this is a textbook case for why we would want to support
>     dynamic linking. The reference implementation of the Infinote protocol,
>     libinfinote, is released under the LGPL, so it would save a great deal
>     of effort to be able to use that rather than creating a new,
>     independent, under-manned implementation from scratch.
>
> We could link with it non-dynamically too.

With non-dynamic linking it's still possible, but such a project would
be forced to move more slowly since recompiling would be necessary for
every user. If I were able to write something that could use libobby and
distribute it to others without making them patch and recompile Emacs, I
would be much more likely to start such a project. Every so often I read
about extensions to Emacs that require a recompile, but I've never
actually used one since it's more complicated to get started with them;
I just don't bother.

If the project were worked on by core Emacs developers in the official
repository, this would be less of an issue since people are more
familiar with just pulling updates and using that. But it seems
inconsiderate to the community to say something is not important just
because it doesn't bother the core developers.

> I don't understand the structure of the situation.  Did you
> reimplement the same thing in Lisp?

That's correct; I implemented a subset of the protocol in Lisp. But
since it was not documented at the time, I did not implement
transformations, just a naieve edit scheme that assumed there would
never be conflicts. So it worked reasonably well as long as you were on
a local network and latencies were low, but over the Internet you would
see inconsistencies.

-Phil

Re: obby

[Phil Hagelberg]

Christian Lynbech <christian@defun.dk> writes:

>>     I had planned to revisit the problem once the new implementation of the
>>     obby protocol and client (now called Infinote) had been released.
>
> What is the main target: to specifically make a gobby compatible client
> or to more generally just add collaborative editing to emacs?

My idea with Ebby was to make it compatible with gobby since I did not
have the time or knowledge necessary to write a server or design a
protocol. But this may not be the case with others who want to work on
collaborative features.

I feel the network effects of being able to interact with users of other
editors are valuable, but if others feel there are good reasons to start
from scratch I won't protest.

-Phil

Re: obby

[Karl Fogel]

Phil Hagelberg <phil@hagelb.org> writes:
> With non-dynamic linking it's still possible, but such a project would
> be forced to move more slowly since recompiling would be necessary for
> every user. If I were able to write something that could use libobby and
> distribute it to others without making them patch and recompile Emacs, I
> would be much more likely to start such a project. Every so often I read
> about extensions to Emacs that require a recompile, but I've never
> actually used one since it's more complicated to get started with them;
> I just don't bother.
>
> If the project were worked on by core Emacs developers in the official
> repository, this would be less of an issue since people are more
> familiar with just pulling updates and using that. But it seems
> inconsiderate to the community to say something is not important just
> because it doesn't bother the core developers.

You'll still get *some* users, though -- compiling Emacs is only maybe
one notch harder than a standard build procedure.  Whether the project
is worked on by core devs or not shouldn't matter, because it doesn't
affect the need for recompilation during development.  After all, people
can just pull updates from *your* repository too, into their core Emacs
tree, then apply and rebuild there.

Given the choice between reimplementing it from scratch and just using
the existing library statically (at the expense of installation ease),
I'd say use the library, assuming it's license-compatible with Emacs.
The installation ease problem will go away entirely later if this is
shipped with core Emacs, after all.  A from-scratch implementation would
always lag, and that would probably be even more of a disincentive to
beta users than installation difficulty would be!

-Karl

Re: obby

[Richard M Stallman]

    What is the main target: to specifically make a gobby compatible client
    or to more generally just add collaborative editing to emacs?

I think collaborative editing in Emacs is the most desirable thing,
but interoperability with other editors would be nice too.

Re: obby

[Richard M Stallman]

    X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00 autolearn=ham 
	    version=3.1.0
    From: Phil Hagelberg <phil@hagelb.org>
    To: rms@gnu.org
    Subject: Re: obby
    Cc: emacs-devel@gnu.org, jtk@yahoo.com
    Date: Fri, 22 May 2009 09:42:56 -0700
    In-Reply-To: <E1M7D7D-0006Ju-8i@fencepost.gnu.org> (Richard M. Stallman's
	    message of "Thu, 21 May 2009 14:35:39 -0400")
    Message-ID: <87my95w0kv.fsf@hagelb.org>
    MIME-Version: 1.0
    Content-Type: text/plain; charset=us-ascii

    Richard M Stallman <rms@gnu.org> writes:

    >     I had planned to revisit the problem once the new implementation of the
    >     obby protocol and client (now called Infinote) had been released. But
    >     implementing inclusion transformation is very difficult.
    >
    > Can you tell us more about this problem?

    The first part involves keeping a lot of state. Every edit that could be
    re-applied (by someone else sending a conflicting edit to the same
    portion of the document before yours reached the server) would need to
    be stored along with some metadata. (It may be possible to tie into
    Emacs' existing undo mechanism for this part if you were able to add
    some metadata to each action.)

I don't think that would be too hard.

    But then if you are notified of a conflict, you need to un-apply your
    changes, apply the canonical ones you received from the server, and
    transform your edits so they apply cleanly. This is pretty
    complicated.

It sounds like what undo-make-selective-list does.
This might be quite easy.

Re: obby

[Stefan Monnier]

> Given the choice between reimplementing it from scratch and just using
> the existing library statically (at the expense of installation ease),
> I'd say use the library, assuming it's license-compatible with Emacs.
> The installation ease problem will go away entirely later if this is
> shipped with core Emacs, after all.  A from-scratch implementation would
> always lag, and that would probably be even more of a disincentive to
> beta users than installation difficulty would be!

The problem goes a bit further: most distributions, like Debian and
friends, will find it inconvenient to build their Emacs binary against
that library unless that library is small, or it's likely to already be
installed, or most Emacs users are likely to want to use it.
In the case of support for `obby', I don't know the size of the library,
but at least I don't expect most Emacs users to use it, and I don't
expect most installs to already have the library installed.
That's a significant problem for "static linking" (it's actually still
dynamically-linked by the way, altough the list of libraries thus linked
is known statically).


        Stefan

Re: obby

[Karl Fogel]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> The problem goes a bit further: most distributions, like Debian and
> friends, will find it inconvenient to build their Emacs binary against
> that library unless that library is small, or it's likely to already be
> installed, or most Emacs users are likely to want to use it.
> In the case of support for `obby', I don't know the size of the library,
> but at least I don't expect most Emacs users to use it, and I don't
> expect most installs to already have the library installed.

The library is about 2.8M in source release.  It's C++.
(Obtained from http://gobby.0x539.de/trac/wiki/Download)

> That's a significant problem for "static linking" (it's actually still
> dynamically-linked by the way, altough the list of libraries thus linked
> is known statically).

Until the library is widely distributed, we could just ship its source
in Emacs (the same way other programs do with SQLite, for example).
Well, 2.8M is a lot to just support one feature; I'm not sure what our
standards are for decisions like this.

But could we conditionally link against it?  Use it if it is there, and
have Ebby simply return errors otherwise, while the rest of Emacs would
be unaffected?

The reason I'm interested is that Gobby, the program for which the
'obby' protocol was invented, is getting quite popular.  I'm sitting
right now at a conference where it's the conference-wide default for
collaborative editing, and this is not the first time that's happened.
If this protocol is going to become a standard for collaborative editing
(as seems likely, at least from here) then it could be a big win for
Emacs to support it.

I can't devote development time to this myself, so I'm cheering for
anything that makes it easier for Phil Hagelberg to work on it :-).

-Karl

Re: obby

[Stefan Monnier]

> But could we conditionally link against it?  Use it if it is there, and
> have Ebby simply return errors otherwise, while the rest of Emacs would
> be unaffected?

It is true that we could do the same as we do for image libraries
in w32: it gets dynamically loaded at runtime, but only if present, and
the dynamic loading is specific to that library, so it can't be used for
arbitrary non-GPL'd code.
Having a real, generic dynamic-loading solution would be better, but
that would be a good stop-gap.


        Stefan