A new collaborative editing package (maybe tangent)

A new collaborative editing package (maybe tangent)

[Yuan Fu]

Hi,

I’ve been working on a collaborative editing package on-and-off for a
while. I call it collab-mode. It’s far from complete but I want to
gather some early feedback (and find some people to test it).

Yes, another collab editing package. I know there’s a pile of
abandoned collab editing packages already, and whether it’s really
very useful is questionable in the age of google docs. Nevertheless,
there seems to be some niche use-cases and I want to make something
original.

It is still very much a WIP so expect it to break a lot and maybe
don’t post it on Reddit or HN :-) After all, I might never complete
and publish it.

My goal for collab-mode is for it to be a p2p, cross-editor, real-time
collab editing package (Emacs collab with Vim!). It works kind of like
language servers (LSP): a local “collab server” (called collab
process) talks to the editor over JSONRPC, and collab processes talk
among themselves over the internet. Most of the hard work is done by
the collab process and the editor end should be relatively easy to
implement.

As of right now, collab-mode has these features:
- Sharing a plain text file
- Sharing a directory (still have rough edges)
- Undo & redo (linear)
- Builtin NAT traversal

And it lacks:
- Authentication and access control
- Auto-resume from network breakage
- Follow mode
- Better error message and error handling in general
- Documentation for writing a editor frontend
- Any form of API stability whatsoever

And obviously there’s no support for other editors yet.


Getting started
===============

I hope I didn’t scare everyone away, here’s how to try it out:

You can find the project at https://github.com/casouri/collab-mode

You can either clone the repo and

    make session-1

to run the collab process (you need rust toolchain to compile it). Or
download the pre-built binary from Releases [1], and run the server
with

    RUST_BACKTRACE=1 RUST_LOG=debug ./collab-mode run --socket

Then, load lisp/collab-mode.el, and

    M-x collab RET

It’ll prompt for your name, and give you some options to continue.
Press:

  h  to open the dashboard (hub)
  s  to share the current buffer
  S  to share a file or directory

Let’s go to the hub first. The dashboard should look like this:

+-----------------------------------------------------------+
|Connection: UP                                             |
|Connection type: (socket 7701)                             |
|Accepting remote peer connections: NO                      |
|                                                           |
|                                                           |
|                                                           |
|🧚 No shared docs, not here, not now.                      |
|Let’s create one, and here’s how!                          |
|                                                           |
|                                                           |
|PRESS + TO SHARE A FILE                                    |
|PRESS C TO CONNECT TO A REMOTE DOC                         |
|PRESS A TO ACCEPT REMOTE CONNECTIONS (for 180s)            |
|                                                           |
------------------------------------------------------------+

You can share a file by pressing "+". It’ll open that file, start
collab-monitored-mode (which should flash green on success), and
present the share link in the hub:

+-----------------------------------------------------------+
|Connection: UP                                             |
|Connection type: (socket 7701)                             |
|Accepting remote peer connections: YES                     |
|Server at: wss://signal.collab-mode.org/c28c65c2-a02a-11e..|
|                                                           |
|                                                           |
|🧚 Your file is shared, and here’s the link                |
|Anyone can connect with just a click!                      |
|LINK: wss://signal.collab-mode.org/c28c65c2-a02a-11ee-b1e..|
|                                                           |
|                                                           |
|self                                                       |
|shared-file.txt *                                          |
|                                                           |
|PRESS + TO SHARE A FILE                                    |
|PRESS C TO CONNECT TO A REMOTE DOC                         |
|PRESS A TO ACCEPT REMOTE CONNECTIONS (for 180s)            |
|                                                           |
+-----------------------------------------------------------+

You’ll notice that the link starts with signal.collab-mode.org. Does
that mean your file is hosted on that server? No. The sharer hosts the
file. This server is a _signaling_ server that I set up for NAT
traversal, ie, to help setup the connection between collab processes.

On another editor, go to the hub, type "C", and enter the link;
collab-mode will connect to the other editor and open the shared file.
It’ll open the buffer and start collab-monitored-mode (followed by a
green flash on success).

Note that, when you share a file, your local collab process will be
open for public connection for 3 minutes. During this time, since
there’s no authentication right now, anyone that knows your address
(wss://signal.collab-mode.org/c28c65c2-...) can connect to you.

If you want to accept remote connection again, press "A" in the hub.
That’ll give you another 3 minutes.


In the shared buffer
====================

In the shared buffer, you can only use collab-mode’s undo and redo
function, which are bound to "C-/" and "C-.", respectively. You can
also use undo-only and undo-redo, they are remapped to collab-mode’s
undo/redo commands.

In the shared buffer, use collab-share-link to copy the share link to
clipboard.


In the hub buffer
=================

I’ve setup eldoc for the hub buffer. When you leave point on a doc,
eldoc will show you the available commands for this doc.


When things go wrong
====================

When refreshing in the hub buffer (pressing "g"), if something goes
wrong (can’t connect to local or remote collab process), the error
will be printed in the bottom.

When in a shared buffer, if something goes wrong, the screen will
flash red, and Emacs will show a warning detailing the error. In that
case, if you can, please collect the warning and terminal logs and
send them to me, so I can try to debug it.


For the curious minds
=====================

There’s a README in src/README.md, it’s mostly for myself so it’ll be
hard to read for others. And before you ask, collab-mode doesn’t use
CRDT, it uses OT with tombstones.


Help with rhymes
================

I hope you don’t find the fairy and her bad rhymes annoying. I’m not
very good at rhymes, chatgpt is even worse. If you can come up with
better ones, please do send them along :-)


[1] The prebuilt binaries are debug build, which are pretty large; the
release build is much smaller (~6MB).

Re: A new collaborative editing package (maybe tangent)

[Karl Fogel]

On 29 Dec 2023, Yuan Fu wrote:
>I’ve been working on a collaborative editing package on-and-off 
>for a
>while. I call it collab-mode. It’s far from complete but I want 
>to
>gather some early feedback (and find some people to test it).
>
>Yes, another collab editing package. I know there’s a pile of
>abandoned collab editing packages already, and whether it’s 
>really
>very useful is questionable in the age of google docs.

There is no question at all about its usefulness!  Having a 
reliable, usable, and easy-to-install Emacs package for 
collaborative editing would be amazing.  Even if it were to only 
support Emacs<->Emacs it would be a great thing; if it allows 
collaborative editing with people using other editors, then it 
will make the stars shine and the moon sing.

I know you said it's not there yet, but I just wanted to respond 
to your remark about questionable usefulness.  Many people would 
love to have this.  I hope you keep working on it!  (Personally, 
if I had time for sustained Emacs development, this would be my 
highest priority.)

>Nevertheless, there seems to be some niche use-cases and I want
>to make something original.

Not niche, not niche! :-)

I'll try to test it with a colleague in January.

Since you're using OT anyway, I wonder if eventually it can be 
compatible with Etherpad, which I believe is also OT not CRDT. 
That would be a nice win.

>It is still very much a WIP so expect it to break a lot and maybe
>don’t post it on Reddit or HN :-) After all, I might never 
>complete
>and publish it.

Understood.

Good luck!  We'll report bugs in your tracker when ready.

Best regards,
-Karl

>My goal for collab-mode is for it to be a p2p, cross-editor, 
>real-time
>collab editing package (Emacs collab with Vim!). It works kind of 
>like
>language servers (LSP): a local “collab server” (called collab
>process) talks to the editor over JSONRPC, and collab processes 
>talk
>among themselves over the internet. Most of the hard work is done 
>by
>the collab process and the editor end should be relatively easy 
>to
>implement.
>
>As of right now, collab-mode has these features:
>- Sharing a plain text file
>- Sharing a directory (still have rough edges)
>- Undo & redo (linear)
>- Builtin NAT traversal
>
>And it lacks:
>- Authentication and access control
>- Auto-resume from network breakage
>- Follow mode
>- Better error message and error handling in general
>- Documentation for writing a editor frontend
>- Any form of API stability whatsoever
>
>And obviously there’s no support for other editors yet.
>
>
>Getting started
>===============
>
>I hope I didn’t scare everyone away, here’s how to try it out:
>
>You can find the project at 
>https://github.com/casouri/collab-mode
>
>You can either clone the repo and
>
>    make session-1
>
>to run the collab process (you need rust toolchain to compile 
>it). Or
>download the pre-built binary from Releases [1], and run the 
>server
>with
>
>    RUST_BACKTRACE=1 RUST_LOG=debug ./collab-mode run --socket
>
>Then, load lisp/collab-mode.el, and
>
>    M-x collab RET
>
>It’ll prompt for your name, and give you some options to 
>continue.
>Press:
>
>  h  to open the dashboard (hub)
>  s  to share the current buffer
>  S  to share a file or directory
>
>Let’s go to the hub first. The dashboard should look like this:
>
>+-----------------------------------------------------------+
>|Connection: UP                                             |
>|Connection type: (socket 7701)                             |
>|Accepting remote peer connections: NO                      |
>|                                                           |
>|                                                           |
>|                                                           |
>|🧚 No shared docs, not here, not now.                      |
>|Let’s create one, and here’s how!                          |
>|                                                           |
>|                                                           |
>|PRESS + TO SHARE A FILE                                    |
>|PRESS C TO CONNECT TO A REMOTE DOC                         |
>|PRESS A TO ACCEPT REMOTE CONNECTIONS (for 180s)            |
>|                                                           |
>------------------------------------------------------------+
>
>You can share a file by pressing "+". It’ll open that file, start
>collab-monitored-mode (which should flash green on success), and
>present the share link in the hub:
>
>+-----------------------------------------------------------+
>|Connection: UP                                             |
>|Connection type: (socket 7701)                             |
>|Accepting remote peer connections: YES                     |
>|Server at: wss://signal.collab-mode.org/c28c65c2-a02a-11e..|
>|                                                           |
>|                                                           |
>|🧚 Your file is shared, and here’s the link                |
>|Anyone can connect with just a click!                      |
>|LINK: wss://signal.collab-mode.org/c28c65c2-a02a-11ee-b1e..|
>|                                                           |
>|                                                           |
>|self                                                       |
>|shared-file.txt *                                          |
>|                                                           |
>|PRESS + TO SHARE A FILE                                    |
>|PRESS C TO CONNECT TO A REMOTE DOC                         |
>|PRESS A TO ACCEPT REMOTE CONNECTIONS (for 180s)            |
>|                                                           |
>+-----------------------------------------------------------+
>
>You’ll notice that the link starts with 
>signal.collab-mode.org. Does
>that mean your file is hosted on that server? No. The sharer 
>hosts the
>file. This server is a _signaling_ server that I set up for NAT
>traversal, ie, to help setup the connection between collab 
>processes.
>
>On another editor, go to the hub, type "C", and enter the link;
>collab-mode will connect to the other editor and open the shared 
>file.
>It’ll open the buffer and start collab-monitored-mode (followed 
>by a
>green flash on success).
>
>Note that, when you share a file, your local collab process will 
>be
>open for public connection for 3 minutes. During this time, since
>there’s no authentication right now, anyone that knows your 
>address
>(wss://signal.collab-mode.org/c28c65c2-...) can connect to you.
>
>If you want to accept remote connection again, press "A" in the 
>hub.
>That’ll give you another 3 minutes.
>
>
>In the shared buffer
>====================
>
>In the shared buffer, you can only use collab-mode’s undo and 
>redo
>function, which are bound to "C-/" and "C-.", respectively. You 
>can
>also use undo-only and undo-redo, they are remapped to 
>collab-mode’s
>undo/redo commands.
>
>In the shared buffer, use collab-share-link to copy the share 
>link to
>clipboard.
>
>
>In the hub buffer
>=================
>
>I’ve setup eldoc for the hub buffer. When you leave point on a 
>doc,
>eldoc will show you the available commands for this doc.
>
>
>When things go wrong
>====================
>
>When refreshing in the hub buffer (pressing "g"), if something 
>goes
>wrong (can’t connect to local or remote collab process), the 
>error
>will be printed in the bottom.
>
>When in a shared buffer, if something goes wrong, the screen will
>flash red, and Emacs will show a warning detailing the error. In 
>that
>case, if you can, please collect the warning and terminal logs 
>and
>send them to me, so I can try to debug it.
>
>
>For the curious minds
>=====================
>
>There’s a README in src/README.md, it’s mostly for myself so 
>it’ll be
>hard to read for others. And before you ask, collab-mode doesn’t 
>use
>CRDT, it uses OT with tombstones.
>
>
>Help with rhymes
>================
>
>I hope you don’t find the fairy and her bad rhymes annoying. I’m 
>not
>very good at rhymes, chatgpt is even worse. If you can come up 
>with
>better ones, please do send them along :-)
>
>
>[1] The prebuilt binaries are debug build, which are pretty 
>large; the
>release build is much smaller (~6MB).

Aw: A new collaborative editing package (maybe tangent)

[Arne Babenhauserheide]

Yuan Fu wrote:
> My goal for collab-mode is for it to be a p2p, cross-editor, real-time
> collab editing package (Emacs collab with Vim!). It works kind of like
> language servers (LSP): a local “collab server” (called collab
> process) talks to the editor over JSONRPC, and collab processes talk
> among themselves over the internet. Most of the hard work is done by
> the collab process and the editor end should be relatively easy to
> implement.

This sounds pretty interesting!

> As of right now, collab-mode has these features:
> - Sharing a plain text file
> - Sharing a directory (still have rough edges)

Does it work well with editing modes? Plain text can be a loose definition with Emacs :-)

Example: When I run a plantuml code block in an org file in a shared directory, will the other side see the generated diagram inline?

> - Undo & redo (linear)
> - Builtin NAT traversal

How do you achieve NAT traversal?
(that's interesting to me from a technical / sharing techniques viewpoint; would love to pass it on to students in a lecture in spring)

I've seen that you use a signaling server, does this manage to enable direct connections between two NAT'ed computers?

> |🧚 No shared docs, not here, not now.                      |
> |Let’s create one, and here’s how!                          |

I would suggest "here is how" instead of "and here's how".
That better matches the rhythm of the first line :-)

> |🧚 Your file is shared, and here’s the link                |
> |Anyone can connect with just a click!                      |

Anyone -> All
(also for rhythm)

Though that's less precise, since if "All" tried to connect, they would likely break something.

> Help with rhymes
> ================
> 
> I hope you don’t find the fairy and her bad rhymes annoying. I’m not
> very good at rhymes, chatgpt is even worse. If you can come up with
> better ones, please do send them along :-)

I hope the suggestions help :-)

How could I run a signaling server myself?

Best wishes,
Arne

Re: A new collaborative editing package (maybe tangent)

[Philip Kaludercic]

Karl Fogel <kfogel@red-bean.com> writes:

> On 29 Dec 2023, Yuan Fu wrote:
>> I’ve been working on a collaborative editing package on-and-off for
>> a
>> while. I call it collab-mode. It’s far from complete but I want to
>>gather some early feedback (and find some people to test it).
>>
>>Yes, another collab editing package. I know there’s a pile of
>> abandoned collab editing packages already, and whether it’s really
>>very useful is questionable in the age of google docs.
>
> There is no question at all about its usefulness!  Having a reliable,
> usable, and easy-to-install Emacs package for collaborative editing
> would be amazing.  Even if it were to only support Emacs<->Emacs it
> would be a great thing; if it allows collaborative editing with people
> using other editors, then it will make the stars shine and the moon
> sing.

Just because it doesn't appear to be mentioned, are you familiar with
the CRDT package on GNU ELPA?  My understanding is that it is not
compatible with other editors, due to limits in the API they expose, but
it does provide P2P collaboration.

Re: A new collaborative editing package (maybe tangent)

[Yuan Fu]

> On Dec 29, 2023, at 9:28 PM, Karl Fogel <kfogel@red-bean.com> wrote:
> 
> On 29 Dec 2023, Yuan Fu wrote:
>> I’ve been working on a collaborative editing package on-and-off for a
>> while. I call it collab-mode. It’s far from complete but I want to
>> gather some early feedback (and find some people to test it).
>> 
>> Yes, another collab editing package. I know there’s a pile of
>> abandoned collab editing packages already, and whether it’s really
>> very useful is questionable in the age of google docs.
> 
> There is no question at all about its usefulness!  Having a reliable, usable, and easy-to-install Emacs package for collaborative editing would be amazing.  Even if it were to only support Emacs<->Emacs it would be a great thing; if it allows collaborative editing with people using other editors, then it will make the stars shine and the moon sing.
> 
> I know you said it's not there yet, but I just wanted to respond to your remark about questionable usefulness.  Many people would love to have this.  I hope you keep working on it!  (Personally, if I had time for sustained Emacs development, this would be my highest priority.)

Thank you for your kind words :-)

> 
>> Nevertheless, there seems to be some niche use-cases and I want
>> to make something original.
> 
> Not niche, not niche! :-)
> 
> I'll try to test it with a colleague in January.

That’ll be very helpful!

> 
> Since you're using OT anyway, I wonder if eventually it can be compatible with Etherpad, which I believe is also OT not CRDT. That would be a nice win.

I didn’t look into it but I imagine it’s gonna be difficult.

> 
>> It is still very much a WIP so expect it to break a lot and maybe
>> don’t post it on Reddit or HN :-) After all, I might never complete
>> and publish it.
> 
> Understood.
> 
> Good luck!  We'll report bugs in your tracker when ready.

Thanks.

Yuan

Re: A new collaborative editing package (maybe tangent)

[Yuan Fu]

> On Dec 30, 2023, at 12:56 AM, Arne Babenhauserheide <arne_bab@web.de> wrote:
> 
> Yuan Fu wrote:
>> My goal for collab-mode is for it to be a p2p, cross-editor, real-time
>> collab editing package (Emacs collab with Vim!). It works kind of like
>> language servers (LSP): a local “collab server” (called collab
>> process) talks to the editor over JSONRPC, and collab processes talk
>> among themselves over the internet. Most of the hard work is done by
>> the collab process and the editor end should be relatively easy to
>> implement.
> 
> This sounds pretty interesting!
> 
>> As of right now, collab-mode has these features:
>> - Sharing a plain text file
>> - Sharing a directory (still have rough edges)
> 
> Does it work well with editing modes? Plain text can be a loose definition with Emacs :-)

Plain text as opposed to rich text (which is usually stored in some internal tree structure).

> 
> Example: When I run a plantuml code block in an org file in a shared directory, will the other side see the generated diagram inline?
> 
>> - Undo & redo (linear)
>> - Builtin NAT traversal
> 
> How do you achieve NAT traversal?
> (that's interesting to me from a technical / sharing techniques viewpoint; would love to pass it on to students in a lecture in spring)

I used webrtc. Rust has a native implementation for it, other languages probably have bindings to libwebrtc. Specifically, webrtc uses ICE (and STUN and TURN) protocol for NAT traversal, and webrtc itself defines a data channel abstraction on top of them.

The stack roughly looks like this:

-------------------------------------
Data channel protocol
-------------------------------------
SCTP - congestion and flow control
-------------------------------------
DTLS - security
-------------------------------------
UDP & ICE - transport & NAT traversal
-------------------------------------

> I've seen that you use a signaling server, does this manage to enable direct connections between two NAT'ed computers?

Right.

> 
>> |🧚 No shared docs, not here, not now.                      |
>> |Let’s create one, and here’s how!                          |
> 
> I would suggest "here is how" instead of "and here's how".
> That better matches the rhythm of the first line :-)
> 
>> |🧚 Your file is shared, and here’s the link                |
>> |Anyone can connect with just a click!                      |
> 
> Anyone -> All
> (also for rhythm)
> 
> Though that's less precise, since if "All" tried to connect, they would likely break something.
> 
>> Help with rhymes
>> ================
>> 
>> I hope you don’t find the fairy and her bad rhymes annoying. I’m not
>> very good at rhymes, chatgpt is even worse. If you can come up with
>> better ones, please do send them along :-)
> 
> I hope the suggestions help :-)

Thanks, they help a lot :-)

> 
> How could I run a signaling server myself?

In the repo there are actually two binaries. One is collab-mode, the other is collab-signal, the signaling server.

You can run the signaling server by 

    make signaling

Or equivalently 

    cargo run --bin collab-signal -- run --port <port>

Yuan

Re: A new collaborative editing package (maybe tangent)

[T.V Raman]

I too think a "shared editing" feature would be enormously useful.

As step 0, I suggest we first colectively define what exactly "shared
editing " means, and use that to specifically what we implement vs what
we dont. In that process, we should also identify "feature enablers" vs
"actual features" see some thoughts below.

Shared Editing:

1. Is this "same time" editing , as in peer-programming, or
   "collaborative authoring" as in tools like Google Docs? Note that for
   the most part,  people use collaborative authoring far more than they
   use "same time editing" with existing tools.

2. For collaborative editing, a core platform requirement i some form of
   shared, persistent storage. Git could be good enough if you could
   mask it from the user ...

3. Features such as comments, comment threads are all "features" in my
   opinion and I suspect Emacs has a plethora of tools for doing this
   already once we crisply identify (1) and (2) above.

   
-- 

Re: A new collaborative editing package (maybe tangent)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

This is potentially very useful, but we should choose the design
we are aiming for, not just drift into it.

Since there are no shared documents,
what do these collab servers communicate about?
Why do they need to communicate at all?
What job do they do?

What sort of encryption do they use?
This would not be good to promote if it does not offer privacy.

What formats can documents be stored in?  Does it handle any format
Emacs can look at?

What external programs does this depend on?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: A new collaborative editing package (maybe tangent)

[Yuan Fu]

> On Dec 31, 2023, at 7:33 AM, T.V Raman <raman@google.com> wrote:
> 
> I too think a "shared editing" feature would be enormously useful.
> 
> As step 0, I suggest we first colectively define what exactly "shared
> editing " means, and use that to specifically what we implement vs what
> we dont. In that process, we should also identify "feature enablers" vs
> "actual features" see some thoughts below.
> 
> Shared Editing:
> 
> 1. Is this "same time" editing , as in peer-programming, or
>   "collaborative authoring" as in tools like Google Docs? Note that for
>   the most part,  people use collaborative authoring far more than they
>   use "same time editing" with existing tools.

Good question! And I’ve given it a lot of thought even before implementing the system. I consider collab-mode primarily a “same time” editing tool. I agree that collaborative authoring would be much more useful, but it has an unavoidable requirement—a server that either directly host the documents or at least help with synchronization and communication between nodes working on the same document. The bottom line is, you need a server with considerable resource. I don’t want to write a program that depends on someone hosting such a service. (Now, collab-mode do require a public signaling server for NAT traversal, but I expect the load on the signaling server to be minimal, so $5/month on Linode or aws should cover it.)

You can say that with careful design, we can make some fully distributed system that requires minimal involvement from a server. I agree with that, but I expect such system to be vastly more complicated while providing a degraded service. That leads to my next consideration: since google doc already exists, is free to use, is only a click away, and is very reliable, why would anyone want to use a program that requires some setup, and is slower and less reliable than google docs?

Having said all that, if someone really want a collaborative authoring system, they can open a collab-mode instance and leave their computer powered on. I think this solution beats any clever, intricate distributed system design by a mile :-)

> 
> 2. For collaborative editing, a core platform requirement i some form of
>   shared, persistent storage. Git could be good enough if you could
>   mask it from the user ...

Or the hard disk of a user that’s willing to leave their computer on for the duration of the collaboration.

> 
> 3. Features such as comments, comment threads are all "features" in my
>   opinion and I suspect Emacs has a plethora of tools for doing this
>   already once we crisply identify (1) and (2) above.

Right, features like these can be easily added regardless of the core design, so they don’t really affect the core design.

Yuan

Re: A new collaborative editing package (maybe tangent)

[Yuan Fu]

> On Dec 31, 2023, at 7:32 PM, Richard Stallman <rms@gnu.org> wrote:
> 
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
> This is potentially very useful, but we should choose the design
> we are aiming for, not just drift into it.
> 
> Since there are no shared documents,
> what do these collab servers communicate about?

Edits made by each user (essentially insertions and deletions).

> Why do they need to communicate at all?

So you can see what I inserted in the the shared document, and vice versa.

> What job do they do?

These locally running servers connect to each other and send user’s edits to each other, so everybody get’s each other’s edit, and the document they edit are kept in sync.

> What sort of encryption do they use?

The servers use webrtc for communication, which uses DTLS for encryption. But as I said, collab-mode is not secure right now, adding authentication and access control is on the TODO list.

> This would not be good to promote if it does not offer privacy.
> 
> What formats can documents be stored in?  Does it handle any format
> Emacs can look at?

Since the servers only send edits, the document itself is stored in Emacs.

> What external programs does this depend on?

The collab-mode server is written in Rust and runs locally; editor-specific frontend will be written in each editors scripting language. The signaling server used to establish connection between two servers is also written in Rust; it needs to run on a public-facing server.

Yuan

Re: A new collaborative editing package (maybe tangent)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Since you're using OT anyway, I wonder if eventually it can be 
  > compatible with Etherpad, which I believe is also OT not CRDT. 

Could you explain what OT and CRDT mean?

First of all, are they nouns, or adjectives?
The first line uses OT as a noun, the second as an adjective.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: A new collaborative editing package (maybe tangent)

[Stefan Kangas]

Yuan Fu <casouri@gmail.com> writes:

>> What sort of encryption do they use?
>
> The servers use webrtc for communication, which uses DTLS for
> encryption. But as I said, collab-mode is not secure right now, adding
> authentication and access control is on the TODO list.

Is it E2E?

Re: A new collaborative editing package (maybe tangent)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Just because it doesn't appear to be mentioned, are you familiar with
  > the CRDT package on GNU ELPA?

I'm not -- could you give us a 10 to 20 line summary of what it can do?
It would be useful to compare the two packages and see to what extent
they do and don't overlap.

If it is possible to merge the two, or extend one to have all the
capabilities of the other, that would be a significant improvement,
because we could pay the cost of one packages and get the benefits
of the two.

By the "cost" I mean the complexity of what users need to know.

On general principles, it is much better to recommend one package
to use for all activities in area A, than to recommend two packages
of which someties P is better and soetimes Q is better.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: A new collaborative editing package (maybe tangent)

[Yuan Fu]

> On Jan 1, 2024, at 3:09 PM, Stefan Kangas <stefankangas@gmail.com> wrote:
> 
> Yuan Fu <casouri@gmail.com> writes:
> 
>>> What sort of encryption do they use?
>> 
>> The servers use webrtc for communication, which uses DTLS for
>> encryption. But as I said, collab-mode is not secure right now, adding
>> authentication and access control is on the TODO list.
> 
> Is it E2E?

Yes, if I do everything correctly, which I’m not (as for now).

Yuan

Re: A new collaborative editing package (maybe tangent)

[Yuan Fu]

> On Jan 1, 2024, at 7:49 AM, Richard Stallman <rms@gnu.org> wrote:
> 
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>> Since you're using OT anyway, I wonder if eventually it can be 
>> compatible with Etherpad, which I believe is also OT not CRDT. 
> 
> Could you explain what OT and CRDT mean?
> 
> First of all, are they nouns, or adjectives?
> The first line uses OT as a noun, the second as an adjective.

They are the two main approaches to designing algorithms for collaborative editing. OT stands for operational transformation, CRDT stands for conflict-free replicated data type. They are different enough that when designing a collab editing system, you need to choose one or another.

Yuan

Re: A new collaborative editing package (maybe tangent)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Since there are no shared documents,
  > > what do these collab servers communicate about?

  > Edits made by each user (essentially insertions and deletions).

  > > Why do they need to communicate at all?

  > So you can see what I inserted in the the shared document, and vice versa.

That makes sense, but what puzzled mwthe your statement that it doesn't
support "shared docs."  Now it seems to support them.

What was the intended meaning of that statement?  It seemed to say
that something was not supported -- what was that something?

  > The servers use webrtc for communication, which uses DTLS for
  > encryption. But as I said, collab-mode is not secure right now,
  > adding authentication and access control is on the TODO list.

There's no hurry in getting everything finished.  If these fit into
the design, we can wait for them to be implemented.

What we should do now, however, before installing anything,
is decide what architecture we want to use for collaborative editing.

For instance, is this the right protocol to use?
Do we want to base communication on this protocol?

Should we base access control on logging in to some other sort
of service?  Could we base it on access to a repo which is where
the shared document is to be stored?

We should decide those things now because changing them later
may be difficult.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: A new collaborative editing package (maybe tangent)

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 776 bytes --]


Richard Stallman <rms@gnu.org> writes:
> What we should do now, however, before installing anything,
> is decide what architecture we want to use for collaborative editing.
>
> For instance, is this the right protocol to use?
> Do we want to base communication on this protocol?

The decision for OT seems to already be taken and may not be easy to
change midflight, so I don’t know whether discussing this part has much
merit.

> Should we base access control on logging in to some other sort
> of service?  Could we base it on access to a repo which is where
> the shared document is to be stored?

And which methods are already well-supported in Rust?

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: A new collaborative editing package (maybe tangent)

[Yuan Fu]

> On Jan 4, 2024, at 12:02 AM, Dr. Arne Babenhauserheide <arne_bab@web.de> wrote:
> 
> 
> Richard Stallman <rms@gnu.org> writes:
>> What we should do now, however, before installing anything,
>> is decide what architecture we want to use for collaborative editing.

I don’t think we are installing anything into Emacs. Collab-mode, if ever completes, will be like any other Emacs package.

>> 
>> For instance, is this the right protocol to use?
>> Do we want to base communication on this protocol?
> 
> The decision for OT seems to already be taken and may not be easy to
> change midflight, so I don’t know whether discussing this part has much
> merit.

Also, since what ever protocol collab-mode uses will be limited to collab-mode itself, I don’t think the protocol is very interesting to discuss anyway.

> 
>> Should we base access control on logging in to some other sort
>> of service?  Could we base it on access to a repo which is where
>> the shared document is to be stored?
> 
> And which methods are already well-supported in Rust?

I’ve thought about using public keys for authentication. Users would share their public keys through secure side-channel, then it’s easy to verify you are connecting to the person you think you are connecting to.

I also thought about adding OpenID support to the signaling server, such that the signaling server can endorse each user’s identify. Is is obviously more complicated, and requires the user to trust the signaling server.

Yuan

Re: A new collaborative editing package (maybe tangent)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > They are the two main approaches to designing algorithms for
  > collaborative editing. OT stands for operational transformation,
  > CRDT stands for conflict-free replicated data type. They are
  > different enough that when designing a collab editing system, you
  > need to choose one or another.

It sounds like this is a decision we should make carefully now.
based on the advantages and disadvantages of each.

Are there any other alternatives we should consider?

Would the choice of any one provide real, useful interoperability?

I don't know anything about this area, so I hope that people who do
know will get together and think about this choice.  But we shouldn't
just drift into one choice or the other!

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: A new collaborative editing package (maybe tangent)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I don’t think we are installing anything into Emacs. Collab-mode,
  > if ever completes, will be like any other Emacs package.

They are all installed in Emacs.  If we add a package to Emacs that
choose method A instead of method B, that choice has inertia.  In
principle, we could later decide method B is better and switch to it.
But it can be difficult to switch to mainly using method B, and even
harder to switch entirely.  We would tend to continue in path A
by inertia.

I don't know anythng about these two methods for collaborative
editing, so I have no preference.  I do know that whatever we do about
this will create inertia for the future.  Rather than letting
historical chance decide which course we take, let's make a thoughtful
decision.

Would it make sense for collab-mode to support both methods?  Are the
differences just matters of detail, or are the concepts
incommensurable?

  > Also, since what ever protocol collab-mode uses will be limited to
  > collab-mode itself,

Why so?  There are other collaborative editing systems, right?  Why
can't it be compatible with those that use the same method - or at
least one of them (if they are incompatible with each other)?




-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: A new collaborative editing package (maybe tangent)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I’ve thought about using public keys for authentication. Users
  > would share their public keys through secure side-channel, then
  > it’s easy to verify you are connecting to the person you think you
  > are connecting to.

This has the advantage of being similar to what we already use.

  > I also thought about adding OpenID support to the signaling
  > server, such that the signaling server can endorse each user’s
  > identify. Is is obviously more complicated, and requires the user
  > to trust the signaling server.

ISTR that there was some doubt about whether OpenID could be used
in the free world.  That was discussed maybe 6 or more years ago.
If it is useful, I can look up what was said back then and we could
see whee it stands now.

Would it be feasible to support both methods?




-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: A new collaborative editing package (maybe tangent)

[Yuan Fu]

> On Jan 5, 2024, at 8:33 PM, Richard Stallman <rms@gnu.org> wrote:
> 
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>> I don’t think we are installing anything into Emacs. Collab-mode,
>> if ever completes, will be like any other Emacs package.
> 
> They are all installed in Emacs.  If we add a package to Emacs that
> choose method A instead of method B, that choice has inertia.  In
> principle, we could later decide method B is better and switch to it.
> But it can be difficult to switch to mainly using method B, and even
> harder to switch entirely.  We would tend to continue in path A
> by inertia.
> 
> I don't know anythng about these two methods for collaborative
> editing, so I have no preference.  I do know that whatever we do about
> this will create inertia for the future.  Rather than letting
> historical chance decide which course we take, let's make a thoughtful
> decision.
> 
> Would it make sense for collab-mode to support both methods?  Are the
> differences just matters of detail, or are the concepts
> incommensurable?

It wouldn’t make sense. As I said, they are significantly different. 

> 
>> Also, since what ever protocol collab-mode uses will be limited to
>> collab-mode itself,
> 
> Why so?  There are other collaborative editing systems, right?  Why
> can't it be compatible with those that use the same method - or at
> least one of them (if they are incompatible with each other)?

I don’t think any one of them are designed with interoperability in mind; most of them are non-free software. Interoperability isn’t my priority either. My priority is to make collab-mode feature-complete, stable, and secure.

Yuan

Re: A new collaborative editing package (maybe tangent)

[Yuan Fu]

> On Jan 5, 2024, at 8:33 PM, Richard Stallman <rms@gnu.org> wrote:
> 
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>> I’ve thought about using public keys for authentication. Users
>> would share their public keys through secure side-channel, then
>> it’s easy to verify you are connecting to the person you think you
>> are connecting to.
> 
> This has the advantage of being similar to what we already use.
> 
>> I also thought about adding OpenID support to the signaling
>> server, such that the signaling server can endorse each user’s
>> identify. Is is obviously more complicated, and requires the user
>> to trust the signaling server.
> 
> ISTR that there was some doubt about whether OpenID could be used
> in the free world.  That was discussed maybe 6 or more years ago.
> If it is useful, I can look up what was said back then and we could
> see whee it stands now.
> 

Since OpenID is a protocol, what make it unsuitable for the free world?

> Would it be feasible to support both methods?

We’ll see. I haven’t thought too deeply into it.

Yuan

Re: A new collaborative editing package (maybe tangent)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Should we base access control on logging in to some other sort
  > > of service?  Could we base it on access to a repo which is where
  > > the shared document is to be stored?

  > And which methods are already well-supported in Rust?

Please do not assume we will or should use Rust.
The trademark license of Rust raises a complex issue of its own.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: A new collaborative editing package (maybe tangent)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > For instance, is this the right protocol to use?
  > > Do we want to base communication on this protocol?

  > The decision for OT seems to already be taken and may not be easy to
  > change midflight, 

We have not made any decision about whether Emacs should use OT or
CRDG.  Each option is still open, and so _perhaps_ is the option of
supporting both.

However, your response is an example of what I have called "inertia"
-- it illustrates why we should decide carefully about that issue
before taking steps that would tend to turn into a commitment.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: A new collaborative editing package (maybe tangent)

[Yuan Fu]

> On Jan 6, 2024, at 8:28 PM, Richard Stallman <rms@gnu.org> wrote:
> 
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>>> For instance, is this the right protocol to use?
>>> Do we want to base communication on this protocol?
> 
>> The decision for OT seems to already be taken and may not be easy to
>> change midflight, 
> 
> We have not made any decision about whether Emacs should use OT or
> CRDG.  Each option is still open, and so _perhaps_ is the option of
> supporting both.
> 
> However, your response is an example of what I have called "inertia"
> -- it illustrates why we should decide carefully about that issue
> before taking steps that would tend to turn into a commitment.

We are not deciding on what _Emacs_ use, are we? We’ve been merely talking about collab-mode, a package. Also, I’ve already implemented the core features and there’s no way to switch design at this point.

Yuan

Re: A new collaborative editing package (maybe tangent)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Since OpenID is a protocol, what make it unsuitable for the free world?

It is no use asking me that question!  I am no expert on it.  The
point is that we DON'T KNOW whether OpenID can be used in the free world,
or what problems its use may cause.

We can't assume that conclusion simply because it is a protocol!

  > > Would it be feasible to support both methods?

  > We’ll see. I haven’t thought too deeply into it.

Would someone please look at the question now?  That question
is fundamental here.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: A new collaborative editing package (maybe tangent)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Why so?  There are other collaborative editing systems, right?  Why
  > > can't it be compatible with those that use the same method - or at
  > > least one of them (if they are incompatible with each other)?

  > I don’t think any one of them are designed with interoperability
  > in mind; most of them are non-free software.

You may be right about that, but it doesn't conclusively answer
the question.

Are you saying that there is NO other implementation of either OT or CRDG
with which we could feasibly interoperate?  It sounds that way, but I'd like
to be sure I understand.

If interoperability is not a feasible option, that is unfortunate, but
it follows we don't need to think about the possibility of
interoperation with existing systems when designing this.

We still need to think carefully about the factors that do apply,
when choosing whic one, or which ones, of OT and CRDG to support.

  > > Would it make sense for collab-mode to support both methods?  Are the
  > > differences just matters of detail, or are the concepts
  > > incommensurable?

  > It wouldn’t make sense. As I said, they are significantly different. 

I take your word for it that they are significantly different -- but
that's not a very clear or specific statement.  It doesn't
automatically follow that one program cannot support both methods to
some extent.

I suggest raising the question in this form: if I wanted to support
both, how would I go about it?

Would it make sense for machine A, which is hosting a document, to
communicate with machines B and C using OT, and in parallel to
communicate with machines D, E and F using CRDG?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: A new collaborative editing package (maybe tangent)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I should clarify that the idea of supporting both OT and CRDG is a
long-term goal.  If it is basically feasible, our concern now should
be not to preclude doing it eventually.  For right now, that means we
should design collab-mode such that either method could be implemented.

I'm not suggesting we implement them both now.  For now, one method
working is enough.

My idea of supporting both methods was based on supposing that each
would be implemented in various other programs and we would want to
interoperate with all of them.  But if there aren't any other programs
to interoperate with, that goal is only a potential future goal.
There's no practical point in actually implementing the other method
unless/until there is something to interoperate with that way.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)