Question collaborative editing.

Question collaborative editing.

[Ergus]

Hi:

Just a question. I have seen in emacswiki the different alternatives for
collaborative editing and so far all of them look a bit hack so I
decided to ask if anyone is working in such a functionality for
emacs or have any hint. Maybe with tramp or emacsserver?

Have this been discussed before?

Best,
Ergus

Re: Question collaborative editing.

[Stefan Monnier]

> Just a question. I have seen in emacswiki the different alternatives for
> collaborative editing and so far all of them look a bit hack so I
> decided to ask if anyone is working in such a functionality for
> emacs or have any hint. Maybe with tramp or emacsserver?

Discussed before.
Emacsserver/client is definitely not a good solution for it.

Rudel (in GNU ELPA) was a serious attempt to do it, but it's been
unmaintained for a long time, so it needs a fair bit of love.

I don't think there's a protocol for that with a good Emacs client (or
server) yet :-(


        Stefan

Re: Question collaborative editing.

[Ergus]

On Mon, Sep 21, 2020 at 09:35:13AM -0400, Stefan Monnier wrote:
>> Just a question. I have seen in emacswiki the different alternatives for
>> collaborative editing and so far all of them look a bit hack so I
>> decided to ask if anyone is working in such a functionality for
>> emacs or have any hint. Maybe with tramp or emacsserver?
>
>Discussed before.
>Emacsserver/client is definitely not a good solution for it.
>
>Rudel (in GNU ELPA) was a serious attempt to do it, but it's been
>unmaintained for a long time, so it needs a fair bit of love.
>
>I don't think there's a protocol for that with a good Emacs client (or
>server) yet :-(
>
>
>        Stefan
>
Does GNU has any server/protocol for that? a WIP? anything?

I see that there is Tandem where this was discused:
https://github.com/typeintandem/tandem/issues/125

But I don't know the current status. It was an attempt for an emacs
plugin:

https://github.com/jscheid/tandem-emacs

But it required many external dependencies (node.js and python) and also
where some license issues.

Re: Question collaborative editing.

[Karl Fogel]

On 21 Sep 2020, Stefan Monnier wrote:
>> Just a question. I have seen in emacswiki the different alternatives for
>> collaborative editing and so far all of them look a bit hack so I
>> decided to ask if anyone is working in such a functionality for
>> emacs or have any hint. Maybe with tramp or emacsserver?
>
>Discussed before.
>Emacsserver/client is definitely not a good solution for it.
>
>Rudel (in GNU ELPA) was a serious attempt to do it, but it's been
>unmaintained for a long time, so it needs a fair bit of love.
>
>I don't think there's a protocol for that with a good Emacs client (or
>server) yet :-(

I'm very interested in this too, but don't have time to work on it.

However, I've been collecting references related to collaborative editing, especially in Emacs, in case I were ever to get time to work on it.  Perhaps those references would be useful for you, Ergus, so here they are in Org Mode format:

* Collaborative Editing | Parallel Editing | OT, CDRT, and more
** [[https://github.com/holtzermann17/linepad][Linepad]] Emacs + Etherpad.  But no progress since 2012?
** [[https://anarc.at/blog/2018-06-26-collaborative-editors-history/][Historical inventory of collaborative editors]] (2018)
** [[https://github.com/zk-phi/togetherly/][Togetherly]] Emacs-only; simple, but apparently works.  Insecure.  (2017)
** [[https://blog.kevinjahns.de/are-crdts-suitable-for-shared-editing/][Are CRDTs suitable for shared editing?]]
   Author of [[https://github.com/yjs/yjs][Yjs]] talks about his optimizations.
   "In this article, I will introduce you to a simple optimization for
   CRDTs and examine the exact performance trade-off of using Yjs for
   shared editing. I hope to convince you that the overhead is
   actually very small even for large documents with long edit
   histories."
** https://github.com/codimd/server
** [[https://floobits.com/help/plugins/emacs][Floobits client-side Emacs plugin]]
   The server-side is proprietary (see https://floobits.com/) but the Emacs
   client-side plugin is free software, under the Apache-2.0 license; you can do
   'git clone https://github.com/Floobits/floobits-emacs.git' to get it.
   This doesn't work without the proprietary server, of course, but it might help
   as a guide to what's needed on the client side.  I emailed Floobits and asked them
   whether they would consider freeing their server-side code, perhaps for a fee,
   but never received a response.  Email reference:
   > From: Karl Fogel
   > To: Floobits Info Contact Address
   > Subject: An odd business inquiry.
   > Date: Wed, 06 May 2020 00:00:21 -0500
   > Message-ID: <87368d1wuy.fsf@red-bean.com>
*** [[https://github.com/codimd/server/blob/master/docs/dev/ot.md][Docs on Operational Transform]]
*** https://operational-transformation.github.io/
** [[https://martin.kleppmann.com/papers/pushpin-papoc20.pdf][PushPin: Towards Production-Quality Peer-to-PeerCollaboration]]
   Peter van Hardenberg
   Ink & Switch, LLCSan Francisco, CA, USA

   Martin Kleppmann
   University of Cambridge
   Cambridge, United Kingdom
** [[https://lord.io/blog/2019/splicing-crdts/][Notes on Splicing CRDTs for Structured Hypertext]]
** https://github.com/typeintandem/tandem
** "Creating a Collaborative Editor" (Pierre Hedkvist, 11 June 2019)
   https://www.pierrehedkvist.com/posts/1-creating-a-collaborative-editor
** "Data Laced with History: Causal Trees & Operational CRDTs"
   Etherpad, Google Docs, etc
   http://archagon.net/blog/2018/03/24/data-laced-with-history/

Re: Question collaborative editing.

[Ergus]

On Mon, Sep 21, 2020 at 03:25:38PM -0500, Karl Fogel wrote:
>On 21 Sep 2020, Stefan Monnier wrote:
>>> Just a question. I have seen in emacswiki the different alternatives for
>>> collaborative editing and so far all of them look a bit hack so I
>>> decided to ask if anyone is working in such a functionality for
>>> emacs or have any hint. Maybe with tramp or emacsserver?
>>
>>Discussed before.
>>Emacsserver/client is definitely not a good solution for it.
>>
>>Rudel (in GNU ELPA) was a serious attempt to do it, but it's been
>>unmaintained for a long time, so it needs a fair bit of love.
>>
>>I don't think there's a protocol for that with a good Emacs client (or
>>server) yet :-(
>
>I'm very interested in this too, but don't have time to work on it.
>
>However, I've been collecting references related to collaborative editing, especially in Emacs, in case I were ever to get time to work on it.  Perhaps those references would be useful for you, Ergus, so here they are in Org Mode format:
>
>* Collaborative Editing | Parallel Editing | OT, CDRT, and more
>** [[https://github.com/holtzermann17/linepad][Linepad]] Emacs + Etherpad.  But no progress since 2012?
>** [[https://anarc.at/blog/2018-06-26-collaborative-editors-history/][Historical inventory of collaborative editors]] (2018)
>** [[https://github.com/zk-phi/togetherly/][Togetherly]] Emacs-only; simple, but apparently works.  Insecure.  (2017)
>** [[https://blog.kevinjahns.de/are-crdts-suitable-for-shared-editing/][Are CRDTs suitable for shared editing?]]
>   Author of [[https://github.com/yjs/yjs][Yjs]] talks about his optimizations.
>   "In this article, I will introduce you to a simple optimization for
>   CRDTs and examine the exact performance trade-off of using Yjs for
>   shared editing. I hope to convince you that the overhead is
>   actually very small even for large documents with long edit
>   histories."
>** https://github.com/codimd/server
>** [[https://floobits.com/help/plugins/emacs][Floobits client-side Emacs plugin]]
>   The server-side is proprietary (see https://floobits.com/) but the Emacs
>   client-side plugin is free software, under the Apache-2.0 license; you can do
>   'git clone https://github.com/Floobits/floobits-emacs.git' to get it.
>   This doesn't work without the proprietary server, of course, but it might help
>   as a guide to what's needed on the client side.  I emailed Floobits and asked them
>   whether they would consider freeing their server-side code, perhaps for a fee,
>   but never received a response.  Email reference:
>   > From: Karl Fogel
>   > To: Floobits Info Contact Address
>   > Subject: An odd business inquiry.
>   > Date: Wed, 06 May 2020 00:00:21 -0500
>   > Message-ID: <87368d1wuy.fsf@red-bean.com>
>*** [[https://github.com/codimd/server/blob/master/docs/dev/ot.md][Docs on Operational Transform]]
>*** https://operational-transformation.github.io/
>** [[https://martin.kleppmann.com/papers/pushpin-papoc20.pdf][PushPin: Towards Production-Quality Peer-to-PeerCollaboration]]
>   Peter van Hardenberg
>   Ink & Switch, LLCSan Francisco, CA, USA
>
>   Martin Kleppmann
>   University of Cambridge
>   Cambridge, United Kingdom
>** [[https://lord.io/blog/2019/splicing-crdts/][Notes on Splicing CRDTs for Structured Hypertext]]
>** https://github.com/typeintandem/tandem
>** "Creating a Collaborative Editor" (Pierre Hedkvist, 11 June 2019)
>   https://www.pierrehedkvist.com/posts/1-creating-a-collaborative-editor
>** "Data Laced with History: Causal Trees & Operational CRDTs"
>   Etherpad, Google Docs, etc
>   http://archagon.net/blog/2018/03/24/data-laced-with-history/
>

Very thanks Karl thanks to your link I found this:

https://conclave-team.github.io/conclave-site/


Which is indeed very detailed.

Re: Question collaborative editing.

[Fermin]

[-- Attachment #1: Type: text/plain, Size: 4287 bytes --]

This is a great feature for Emacs, thanks to show me the existence of Rudel, maybe the easiest aproach is to make it work with 
https://github.com/gobby/libinfinit

Seems quite active, and the client is in a unstable state.What so you guys think ?

Regards.

On 24 September 2020 03:36:55 CEST, Ergus <spacibba@aol.com> wrote:
>On Mon, Sep 21, 2020 at 03:25:38PM -0500, Karl Fogel wrote:
>>On 21 Sep 2020, Stefan Monnier wrote:
>>>> Just a question. I have seen in emacswiki the different
>alternatives for
>>>> collaborative editing and so far all of them look a bit hack so I
>>>> decided to ask if anyone is working in such a functionality for
>>>> emacs or have any hint. Maybe with tramp or emacsserver?
>>>
>>>Discussed before.
>>>Emacsserver/client is definitely not a good solution for it.
>>>
>>>Rudel (in GNU ELPA) was a serious attempt to do it, but it's been
>>>unmaintained for a long time, so it needs a fair bit of love.
>>>
>>>I don't think there's a protocol for that with a good Emacs client
>(or
>>>server) yet :-(
>>
>>I'm very interested in this too, but don't have time to work on it.
>>
>>However, I've been collecting references related to collaborative
>editing, especially in Emacs, in case I were ever to get time to work
>on it.  Perhaps those references would be useful for you, Ergus, so
>here they are in Org Mode format:
>>
>>* Collaborative Editing | Parallel Editing | OT, CDRT, and more
>>** [[https://github.com/holtzermann17/linepad][Linepad]] Emacs +
>Etherpad.  But no progress since 2012?
>>**
>[[https://anarc.at/blog/2018-06-26-collaborative-editors-history/][Historical
>inventory of collaborative editors]] (2018)
>>** [[https://github.com/zk-phi/togetherly/][Togetherly]] Emacs-only;
>simple, but apparently works.  Insecure.  (2017)
>>**
>[[https://blog.kevinjahns.de/are-crdts-suitable-for-shared-editing/][Are
>CRDTs suitable for shared editing?]]
>>   Author of [[https://github.com/yjs/yjs][Yjs]] talks about his
>optimizations.
>>   "In this article, I will introduce you to a simple optimization for
>>   CRDTs and examine the exact performance trade-off of using Yjs for
>>   shared editing. I hope to convince you that the overhead is
>>   actually very small even for large documents with long edit
>>   histories."
>>** https://github.com/codimd/server
>>** [[https://floobits.com/help/plugins/emacs][Floobits client-side
>Emacs plugin]]
>>   The server-side is proprietary (see https://floobits.com/) but the
>Emacs
>>   client-side plugin is free software, under the Apache-2.0 license;
>you can do
>>   'git clone https://github.com/Floobits/floobits-emacs.git' to get
>it.
>>   This doesn't work without the proprietary server, of course, but it
>might help
>>   as a guide to what's needed on the client side.  I emailed Floobits
>and asked them
>>   whether they would consider freeing their server-side code, perhaps
>for a fee,
>>   but never received a response.  Email reference:
>>   > From: Karl Fogel
>>   > To: Floobits Info Contact Address
>>   > Subject: An odd business inquiry.
>>   > Date: Wed, 06 May 2020 00:00:21 -0500
>>   > Message-ID: <87368d1wuy.fsf@red-bean.com>
>>***
>[[https://github.com/codimd/server/blob/master/docs/dev/ot.md][Docs on
>Operational Transform]]
>>*** https://operational-transformation.github.io/
>>** [[https://martin.kleppmann.com/papers/pushpin-papoc20.pdf][PushPin:
>Towards Production-Quality Peer-to-PeerCollaboration]]
>>   Peter van Hardenberg
>>   Ink & Switch, LLCSan Francisco, CA, USA
>>
>>   Martin Kleppmann
>>   University of Cambridge
>>   Cambridge, United Kingdom
>>** [[https://lord.io/blog/2019/splicing-crdts/][Notes on Splicing
>CRDTs for Structured Hypertext]]
>>** https://github.com/typeintandem/tandem
>>** "Creating a Collaborative Editor" (Pierre Hedkvist, 11 June 2019)
>>  
>https://www.pierrehedkvist.com/posts/1-creating-a-collaborative-editor
>>** "Data Laced with History: Causal Trees & Operational CRDTs"
>>   Etherpad, Google Docs, etc
>>   http://archagon.net/blog/2018/03/24/data-laced-with-history/
>>
>
>Very thanks Karl thanks to your link I found this:
>
>https://conclave-team.github.io/conclave-site/
>
>
>Which is indeed very detailed.

[-- Attachment #2: Type: text/html, Size: 5531 bytes --]

Re: Question collaborative editing.

[Ergus]

On Thu, Sep 24, 2020 at 11:41:27PM +0200, Fermin wrote:
>This is a great feature for Emacs, thanks to show me the existence of Rudel, maybe the easiest aproach is to make it work with
>https://github.com/gobby/libinfinit
>
>Seems quite active, and the client is in a unstable state.What so you guys think ?
>
>Regards.
>
Hi I am trying to find documentation about this and it seems there is
not too much available around about libinfinity; but it is installed in
my system, so it is a "stable" package somehow.

The only issue I see so far is that it is oriented to gtk applications
which could be a problem for practical uses.

OTOH I would prefer something less centralized with a p2p like conclave;
but I am open to any idea that works.

There is also Teletype and Tandem... the first for Atom and the other is
supposed to work with Sublime, Neovim and Apache. Which is very
attractive IMO because somehow will break our own burble to interact
with the rest of the world... But they have a node.js client-server and
I am not sure how efficient will be to use that requiring python3 too.

>On 24 September 2020 03:36:55 CEST, Ergus <spacibba@aol.com> wrote:

Re: Question collaborative editing.

[Fermin]

[-- Attachment #1: Type: text/plain, Size: 1477 bytes --]

Seems like conclave is no longer maintained.
What so you think about putting encryption in the default Rudel sever?

It seems that works well, but I don't know how hard this can be.


On 25 September 2020 02:22:39 CEST, Ergus <spacibba@aol.com> wrote:
>On Thu, Sep 24, 2020 at 11:41:27PM +0200, Fermin wrote:
>>This is a great feature for Emacs, thanks to show me the existence of
>Rudel, maybe the easiest aproach is to make it work with
>>https://github.com/gobby/libinfinit
>>
>>Seems quite active, and the client is in a unstable state.What so you
>guys think ?
>>
>>Regards.
>>
>Hi I am trying to find documentation about this and it seems there is
>not too much available around about libinfinity; but it is installed in
>my system, so it is a "stable" package somehow.
>
>The only issue I see so far is that it is oriented to gtk applications
>which could be a problem for practical uses.
>
>OTOH I would prefer something less centralized with a p2p like
>conclave;
>but I am open to any idea that works.
>
>There is also Teletype and Tandem... the first for Atom and the other
>is
>supposed to work with Sublime, Neovim and Apache. Which is very
>attractive IMO because somehow will break our own burble to interact
>with the rest of the world... But they have a node.js client-server and
>I am not sure how efficient will be to use that requiring python3 too.
>
>>On 24 September 2020 03:36:55 CEST, Ergus <spacibba@aol.com> wrote:

[-- Attachment #2: Type: text/html, Size: 2064 bytes --]

Re: Question collaborative editing.

[Ergus]

On Fri, Sep 25, 2020 at 12:34:27PM +0200, Fermin wrote:
>Seems like conclave is no longer maintained.
>What so you think about putting encryption in the default Rudel sever?
>
Non libinfinity, Teletype, Tandem or any other seem to be very active
these days... which is making me to reconsider if invest time on this
worth it... as... why are the other approaches abandon not only in
emacs. Maybe the feature is not as useful as I think on it...
>
>It seems that works well, but I don't know how hard this can be.
>
I want to try Rudle first to see if it fit some minimal "needs". That
it:

1) Not require modem/routers special configurations to open ports or
have a static IP.

2) Use CRDT instead of OT

3) It could be integrated with other editors to create "mix" sessions. I
mean, it doesn't enforces some emacs specific features and plugins could
be created for other editors.

Said that; Tandem already have all that + it also uses json that have
been improved in emacs recently... BUT requires Python+nodeJS in the
user side... so IMO it is not a no op. I would prefer a simpler C
library/executable as the client that could be integrated with emacs and
distributed for the other editors.

So far the only problem I find with libinfinity is that it totally lacks
documentation.



>
>On 25 September 2020 02:22:39 CEST, Ergus <spacibba@aol.com> wrote:
>>On Thu, Sep 24, 2020 at 11:41:27PM +0200, Fermin wrote:
>>>This is a great feature for Emacs, thanks to show me the existence of
>>Rudel, maybe the easiest aproach is to make it work with
>>>https://github.com/gobby/libinfinit
>>>
>>>Seems quite active, and the client is in a unstable state.What so you
>>guys think ?
>>>
>>>Regards.
>>>
>>Hi I am trying to find documentation about this and it seems there is
>>not too much available around about libinfinity; but it is installed in
>>my system, so it is a "stable" package somehow.
>>
>>The only issue I see so far is that it is oriented to gtk applications
>>which could be a problem for practical uses.
>>
>>OTOH I would prefer something less centralized with a p2p like
>>conclave;
>>but I am open to any idea that works.
>>
>>There is also Teletype and Tandem... the first for Atom and the other
>>is
>>supposed to work with Sublime, Neovim and Apache. Which is very
>>attractive IMO because somehow will break our own burble to interact
>>with the rest of the world... But they have a node.js client-server and
>>I am not sure how efficient will be to use that requiring python3 too.
>>
>>>On 24 September 2020 03:36:55 CEST, Ergus <spacibba@aol.com> wrote:

Re: Question collaborative editing.

[Ergus]

On September 25, 2020 12:34:27 PM GMT+02:00, Fermin <fmfs@posteo.net> wrote:
>Seems like conclave is no longer maintained.
>What so you think about putting encryption in the default Rudel sever?
>
>It seems that works well, but I don't know how hard this can be.
>
Just to record in the thread for future discussions on this topic.


After some emails the author of libinfinity confimed that the project is totally dead and he advices not to use it for future projects anymore.


So IMO rudel is also dead as it relies on libinfinity... unless someone decides to maintain libinfinity. Or create a fork or an alternative.

Best,
Ergus

>On 25 September 2020 02:22:39 CEST, Ergus <spacibba@aol.com> wrote:
>>On Thu, Sep 24, 2020 at 11:41:27PM +0200, Fermin wrote:
>>>This is a great feature for Emacs, thanks to show me the existence of
>>Rudel, maybe the easiest aproach is to make it work with
>>>https://github.com/gobby/libinfinit
>>>
>>>Seems quite active, and the client is in a unstable state.What so you
>>guys think ?
>>>
>>>Regards.
>>>
>>Hi I am trying to find documentation about this and it seems there is
>>not too much available around about libinfinity; but it is installed
>in
>>my system, so it is a "stable" package somehow.
>>
>>The only issue I see so far is that it is oriented to gtk applications
>>which could be a problem for practical uses.
>>
>>OTOH I would prefer something less centralized with a p2p like
>>conclave;
>>but I am open to any idea that works.
>>
>>There is also Teletype and Tandem... the first for Atom and the other
>>is
>>supposed to work with Sublime, Neovim and Apache. Which is very
>>attractive IMO because somehow will break our own burble to interact
>>with the rest of the world... But they have a node.js client-server
>and
>>I am not sure how efficient will be to use that requiring python3 too.
>>
>>>On 24 September 2020 03:36:55 CEST, Ergus <spacibba@aol.com> wrote:

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Question collaborative editing.

[Fermin]

[-- Attachment #1: Type: text/plain, Size: 2373 bytes --]

That's unfortunate, but I think that it may be a little bit of hope, I'm going to try to decouple the rudel native server with the client, and see where I can go.

Thank you.

On 28 September 2020 02:43:27 CEST, Ergus <spacibba@aol.com> wrote:
>
>
>On September 25, 2020 12:34:27 PM GMT+02:00, Fermin <fmfs@posteo.net>
>wrote:
>>Seems like conclave is no longer maintained.
>>What so you think about putting encryption in the default Rudel sever?
>>
>>It seems that works well, but I don't know how hard this can be.
>>
>Just to record in the thread for future discussions on this topic.
>
>
>After some emails the author of libinfinity confimed that the project
>is totally dead and he advices not to use it for future projects
>anymore.
>
>
>So IMO rudel is also dead as it relies on libinfinity... unless someone
>decides to maintain libinfinity. Or create a fork or an alternative.
>
>Best,
>Ergus
>
>>On 25 September 2020 02:22:39 CEST, Ergus <spacibba@aol.com> wrote:
>>>On Thu, Sep 24, 2020 at 11:41:27PM +0200, Fermin wrote:
>>>>This is a great feature for Emacs, thanks to show me the existence
>of
>>>Rudel, maybe the easiest aproach is to make it work with
>>>>https://github.com/gobby/libinfinit
>>>>
>>>>Seems quite active, and the client is in a unstable state.What so
>you
>>>guys think ?
>>>>
>>>>Regards.
>>>>
>>>Hi I am trying to find documentation about this and it seems there is
>>>not too much available around about libinfinity; but it is installed
>>in
>>>my system, so it is a "stable" package somehow.
>>>
>>>The only issue I see so far is that it is oriented to gtk
>applications
>>>which could be a problem for practical uses.
>>>
>>>OTOH I would prefer something less centralized with a p2p like
>>>conclave;
>>>but I am open to any idea that works.
>>>
>>>There is also Teletype and Tandem... the first for Atom and the other
>>>is
>>>supposed to work with Sublime, Neovim and Apache. Which is very
>>>attractive IMO because somehow will break our own burble to interact
>>>with the rest of the world... But they have a node.js client-server
>>and
>>>I am not sure how efficient will be to use that requiring python3
>too.
>>>
>>>>On 24 September 2020 03:36:55 CEST, Ergus <spacibba@aol.com> wrote:
>
>-- 
>Sent from my Android device with K-9 Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 3753 bytes --]

Re: Question collaborative editing.

[Noam Postavsky]

On Mon, 28 Sep 2020 at 20:00, Fermin <fmfs@posteo.net> wrote:
>
> That's unfortunate, but I think that it may be a little bit of hope, I'm going to try to decouple the rudel native server with the client, and see where I can go.

>> So IMO rudel is also dead as it relies on libinfinity... unless someone decides to maintain libinfinity. Or create a fork or an alternative.

As far as I recall, rudel doesn't rely on libinfinity in particular,
that's just one of the possible backends it tries to support.

Re: Question collaborative editing.

[Jean Louis]

Collaboration in Emacs should be native, as that is often used feature by many users of spreadsheets online, many of them use online services and collaborative spreadsheet without two reason for spreadsheet, they want collaboration primarily.

I would like to see something like separate switch out function that launches internet exposed server with carefully selected and authorized files for collaborative editing.

That would give freedom from proprietary software to many users.

Re: Question collaborative editing.

[Ergus]

On Tue, Sep 29, 2020 at 08:25:01AM +0000, Jean Louis wrote:

>Collaboration in Emacs should be native, as that is often used feature
>by many users of spreadsheets online, many of them use online services
>and collaborative spreadsheet without two reason for spreadsheet, they
>want collaboration primarily.
>
>I would like to see something like separate switch out function that
>launches internet exposed server with carefully selected and authorized
>files for collaborative editing.
>
Rudel used to do something like that... Not sure how exactly.

>That would give freedom from proprietary software to many users.

But this has 3 main problems.

1) On one hand such services require some servers (to work like google
spreadsheet) and need to be provided somehow... something difficult as I
don't think gnu or fsf have resources to maintain a service like that
and provide it.

2) On the other hand it will be better if the service is somehow
distributed in order to give more privacy-security but also to reduce
the load of the servers... I still can't find any infrastructure we can
use, cause most of the peer-to-peer libraries are for C++, javascript,
Node.js and so on (example: webrtc). Just on yesterday I found
n2n... But I am not a web specialist so it requires a lot of
experimenting time for me.

3) The other workflow (create a local server for others) is the
"simplest" approach at the moment. But that is a problem for many use
cases due to dynamic ip addreses, firewalls, opening ports and so on. It
is fine for a class room or company, but not for working from home.

Re: Question collaborative editing.

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 509 bytes --]

> 1) On one hand such services require some servers (to work like google
> spreadsheet) and need to be provided somehow... something difficult as I
> don't think gnu or fsf have resources to maintain a service like that
> and provide it.
CRDT should do it. AFAIK the only part that might
require central server is the unique client UID generation but 
we could get rid of it either ask user to input a unique name,
or assume hash doesn’t collide, or implement Paxos in Elisp
the lisp hacker’s way.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing.

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 888 bytes --]

> I want to try Rudle first to see if it fit some minimal "needs". That
> it:
> 
> 1) Not require modem/routers special configurations to open ports or
> have a static IP.
> 
> 2) Use CRDT instead of OT
> 
> 3) It could be integrated with other editors to create "mix" sessions. I
> mean, it doesn't enforces some emacs specific features and plugins could
> be created for other editors.
I took a glance at source of rudel, and I have to admit I didn’t
enjoy reading it — I have to track through several M-. just to
get to some factory method (srsly)? I wonder how much effort
it will take to make such big change in its infrasture.

On the other hand a separate CRDT library is a quite orthogonal
issue and might be potentially reused by other packages, or
even other use cases — like, making Emacs plugins asynchronous.

I’ll work on a separate CRDT library.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing.

[Stefan Monnier]

>> 1) On one hand such services require some servers (to work like google
>> spreadsheet) and need to be provided somehow... something difficult as I
>> don't think gnu or fsf have resources to maintain a service like that
>> and provide it.
> CRDT should do it.

I think CRDT make no difference to the problem referred to here, which
is that two machines, each behind its own firewall will have difficulty
talking to each other unless they go through some external server (at
least to set up the initial connection).  There are *partial* solutions in
the form of ICE/STUN.


        Stefan

Re: Question collaborative editing.

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 386 bytes --]

> I think CRDT make no difference to the problem referred to here, which
> is that two machines, each behind its own firewall will have difficulty
> talking to each other unless they go through some external server (at
> least to set up the initial connection).  There are *partial* solutions in
> the form of ICE/STUN.

I see, or maybe we could run it on existing p2p network e.g. IRC?

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing.

[Karl Fogel]

On 29 Sep 2020, Ergus wrote:
>1) On one hand such services require some servers (to work like google
>spreadsheet) and need to be provided somehow... something difficult as I
>don't think gnu or fsf have resources to maintain a service like that
>and provide it.
>
>2) On the other hand it will be better if the service is somehow
>distributed in order to give more privacy-security but also to reduce
>the load of the servers... I still can't find any infrastructure we can
>use, cause most of the peer-to-peer libraries are for C++, javascript,
>Node.js and so on (example: webrtc). Just on yesterday I found
>n2n... But I am not a web specialist so it requires a lot of
>experimenting time for me.
>
>3) The other workflow (create a local server for others) is the
>"simplest" approach at the moment. But that is a problem for many use
>cases due to dynamic ip addreses, firewalls, opening ports and so on. It
>is fine for a class room or company, but not for working from home.

It's okay if a central server is required, as long as it's free software.  In practice, there would be a few "well-known" central servers that people use (the way many people just use https://pad.riseup.net/ for Etherpad today).  One of those well-known servers can even be set as the default in the client code in Emacs.  If some people want to use a different server instance to collaborate, all they have to do is set that variable, or specify the server through some interactive prompt.

This doesn't necessarily imply OT instead of CRDT or vice-versa.  Still, despite the "CRDTs are the future" message at https://josephg.com/blog/crdts-are-the-future/, I suspect that for Emacs's purposes OT might be the better solution: simpler to implement and maintain.  In any case, I hope development of this feature doesn't block on decentralization.  While decentralization would be nice, it's not required; people can set up their own servers when they really need to.

Best regards,
-Karl

Re: Question collaborative editing.

[Ergus]

On Tue, Sep 29, 2020 at 04:01:56PM +0000, Qiantan Hong wrote:
>> I want to try Rudle first to see if it fit some minimal "needs". That
>> it:
>>
>> 1) Not require modem/routers special configurations to open ports or
>> have a static IP.
>>
>> 2) Use CRDT instead of OT
>>
>> 3) It could be integrated with other editors to create "mix" sessions. I
>> mean, it doesn't enforces some emacs specific features and plugins could
>> be created for other editors.
>I took a glance at source of rudel, and I have to admit I didn’t
>enjoy reading it — I have to track through several M-. just to
>get to some factory method (srsly)? I wonder how much effort
>it will take to make such big change in its infrasture.
>
>On the other hand a separate CRDT library is a quite orthogonal
>issue and might be potentially reused by other packages, or
>even other use cases — like, making Emacs plugins asynchronous.
>
>I’ll work on a separate CRDT library.

Actually this was my intention since the beginning. If you can start
with that is much much better (you can share the link to the repo if you
like and I will try to help).

The basic algorithms could easily be implemented in an editor agnostic C
library. It can receive a json string and return json as well (as we
already have a good and fast json parser in emacs).

Also the communication can he handled in that level. And at the end it
will be possible to implement plugins for almost any editor with json
support or using a json external library....

Did you checked Tandem? It has almost everything we need except because
it is Python+javascript, node.js... :(

You could check these 4 links:

https://github.com/typeintandem/tandem
https://github.com/typeintandem/tandem/issues/125
https://github.com/jscheid/tandem-emacs
https://github.com/typeintandem/tandem/wiki/Interacting-with-the-Tandem-Agent

My idea was more or less to reproduce the last one but without all the
agent, stdin, python and JS mess.

Best,
Ergus

Re: Question collaborative editing.

[Ergus]

On Tue, Sep 29, 2020 at 12:35:54PM -0500, Karl Fogel wrote:
>On 29 Sep 2020, Ergus wrote:
>>1) On one hand such services require some servers (to work like google
>>spreadsheet) and need to be provided somehow... something difficult as I
>>don't think gnu or fsf have resources to maintain a service like that
>>and provide it.
>>
>>2) On the other hand it will be better if the service is somehow
>>distributed in order to give more privacy-security but also to reduce
>>the load of the servers... I still can't find any infrastructure we can
>>use, cause most of the peer-to-peer libraries are for C++, javascript,
>>Node.js and so on (example: webrtc). Just on yesterday I found
>>n2n... But I am not a web specialist so it requires a lot of
>>experimenting time for me.
>>
>>3) The other workflow (create a local server for others) is the
>>"simplest" approach at the moment. But that is a problem for many use
>>cases due to dynamic ip addreses, firewalls, opening ports and so on. It
>>is fine for a class room or company, but not for working from home.
>
>It's okay if a central server is required, as long as it's free software.  In practice, there would be a few "well-known" central servers that people use (the way many people just use https://pad.riseup.net/ for Etherpad today).  One of those well-known servers can even be set as the default in the client code in Emacs.  If some people want to use a different server instance to collaborate, all they have to do is set that variable, or specify the server through some interactive prompt.
>
>This doesn't necessarily imply OT instead of CRDT or vice-versa.  Still, despite the "CRDTs are the future" message at https://josephg.com/blog/crdts-are-the-future/, I suspect that for Emacs's purposes OT might be the better solution: simpler to implement and maintain.  In any case, I hope development of this feature doesn't block on decentralization.  While decentralization would be nice, it's not required; people can set up their own servers when they really need to.
>
>Best regards,
>-Karl
>
Hi Karl:

In the library level we could (and maybe should) implement both (OT and
CRDT). This will only require a hint from the server side (or the
connection starter client) to inform the client's libraries "what to use
this time". Then practice will say what's better.

ATM we could start with a C library in order to provide a nice free
back-end editor agnostic. Because IMO the most interesting part of this
will be to open emacs to interact with other editors with a free
infrastructure emacs-centric.

IMO: it should/must be like Tandem but made in C and using free
libraries. Without python or JS dependencies and with a JSON (or
standard interface).

Re: Question collaborative editing.

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 1108 bytes --]

> In the library level we could (and maybe should) implement both (OT and
> CRDT). This will only require a hint from the server side (or the
> connection starter client) to inform the client's libraries "what to use
> this time". Then practice will say what's better.
> 
> ATM we could start with a C library in order to provide a nice free
> back-end editor agnostic. Because IMO the most interesting part of this
> will be to open emacs to interact with other editors with a free
> infrastructure emacs-centric.

I’m not sure if it’s easy (or worth the effort) to provide CRDT in an 
editor-agnostic way from C-level. Seems that the most natural
choice is to tag characters with CRDT data structures. Emacs has
text property which is very suitable for this, and I’m implementing
it in this way as an Elisp library. However, this relies on the particular
data structure for buffer/text emacs uses.

If implemented as a separate C library, I imagine the CRDT library
need to have its own buffer/text data structure and somehow keep
in sync with Emacs’ .. doesn’t sound so clean.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing.

[Ergus]

On Tue, Sep 29, 2020 at 11:40:53PM +0000, Qiantan Hong wrote:
>> In the library level we could (and maybe should) implement both (OT and
>> CRDT). This will only require a hint from the server side (or the
>> connection starter client) to inform the client's libraries "what to use
>> this time". Then practice will say what's better.
>>
>> ATM we could start with a C library in order to provide a nice free
>> back-end editor agnostic. Because IMO the most interesting part of this
>> will be to open emacs to interact with other editors with a free
>> infrastructure emacs-centric.
>
>I’m not sure if it’s easy (or worth the effort) to provide CRDT in an
>editor-agnostic way from C-level. Seems that the most natural
>choice is to tag characters with CRDT data structures. Emacs has
>text property which is very suitable for this, and I’m implementing
>it in this way as an Elisp library. However, this relies on the particular
>data structure for buffer/text emacs uses.
>
Oh; I get it.

I would be concerned about the performance (but I am paranoid about
Emacs performance anyway so don't worry for that). AFAIR emacs buffers
are stored in a contiguous way (I may be wrong) and in that case the
search with CRDT may be expensive. At least as it was explained here (in
the optimization section):

https://conclave-team.github.io/conclave-site/

Unless some extra optimizations are implemented of course.

>If implemented as a separate C library, I imagine the CRDT library
>need to have its own buffer/text data structure and somehow keep
>in sync with Emacs’ .. doesn’t sound so clean.

Agree. That was exactly my idea. Keep all that in the library and emacs
just processes the final translated outputs with absolute char positions
from the json.

I took Tandem as the starting point in my considerations... But you may
be right. If it is easier to implement and works; it may be fine and
simpler than what I had in mind.

Best,
Ergus

Re: Question collaborative editing.

[Qiantan Hong]

[-- Attachment #1.1: Type: text/plain, Size: 1495 bytes --]

> I would be concerned about the performance (but I am paranoid about
> Emacs performance anyway so don't worry for that). AFAIR emacs buffers
> are stored in a contiguous way (I may be wrong) and in that case the
> search with CRDT may be expensive. At least as it was explained here (in
> the optimization section):
> 
> https://conclave-team.github.io/conclave-site/
> 
> Unless some extra optimizations are implemented of course.
Yes, I’m also concerning that, but nobody knows without trying it
out. At least in principle, text-property-search-* and *-property-change
are C primitives and should be really fast. Also, there’s caching trick
used in Yjs described here 
https://github.com/yjs/yjs/blob/main/INTERNALS.md#item-storage <https://github.com/yjs/yjs/blob/main/INTERNALS.md#item-storage>

On the other hand the text-property/gap-array data structure in
emacs seems to save us from using double linked list, since
the total order is just reflected in the continuous buffer.

In any case I’m afraid its possible to do it for normal size buffer
but it won’t scale for really large buffers — I imagine when buffer
is large enough, the cost of even roaming around the buffer
(possibly using the position cache) to look for place to insert
will be unacceptable, because moving in gap array takes copying
cost proportional to moving distance (I might be wrong)…
But I guess let’s assume people are not going to collaborative edit
some 10MB log file.

[-- Attachment #1.2: Type: text/html, Size: 2145 bytes --]

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing.

[Karl Fogel]

On 30 Sep 2020, Ergus wrote:
>I would be concerned about the performance (but I am paranoid about
>Emacs performance anyway so don't worry for that). AFAIR emacs buffers
>are stored in a contiguous way (I may be wrong)

Information about how Emacs buffers are represented under the hood can be found here:

* https://git.savannah.gnu.org/cgit/emacs.git/tree/src/buffer.h#n213

* https://www.gnu.org/software/emacs/manual/html_node/elisp/Buffer-Gap.html

TL;DR: It's a contiguous array with a gap at the point where insertion or deletion happens.  When you need to insert or delete somewhere else, Emacs first moves the gap to the new location.  The gap is exposed to Elisp via the functions `gap-position' and `gap-size', though I must confess I've never used them from Elisp.

Best regards,
-Karl

Re: Question collaborative editing.

[Jean Louis]

* Ergus <spacibba@aol.com> [2020-09-30 00:47]:
> The basic algorithms could easily be implemented in an editor agnostic C
> library. It can receive a json string and return json as well (as we
> already have a good and fast json parser in emacs).

As it is Emacs, it would be better using Emacs Lisp data, not json.

Jean

Re: Question collaborative editing.

[Ergus]

On Wed, Sep 30, 2020 at 02:40:26AM -0500, Karl Fogel wrote:
>On 30 Sep 2020, Ergus wrote:
>>I would be concerned about the performance (but I am paranoid about
>>Emacs performance anyway so don't worry for that). AFAIR emacs buffers
>>are stored in a contiguous way (I may be wrong)
>
>Information about how Emacs buffers are represented under the hood can be found here:
>
>* https://git.savannah.gnu.org/cgit/emacs.git/tree/src/buffer.h#n213
>
>* https://www.gnu.org/software/emacs/manual/html_node/elisp/Buffer-Gap.html
>
This is interesting indeed.

I was looking at this to compare what's around:

https://ecc-comp.blogspot.com/2015/05/a-brief-glance-at-how-5-text-editors.html

>TL;DR: It's a contiguous array with a gap at the point where insertion or deletion happens.  When you need to insert or delete somewhere else, Emacs first moves the gap to the new location.  The gap is exposed to Elisp via the functions `gap-position' and `gap-size', though I must confess I've never used them from Elisp.
>
>Best regards,
>-Karl

Re: Question collaborative editing.

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> Date: Tue, 29 Sep 2020 23:40:53 +0000
> Cc: Fermin <fmfs@posteo.net>, Jean Louis <bugs@gnu.support>,
>  Noam Postavsky <npostavs@gmail.com>, Emacs developers <emacs-devel@gnu.org>,
>  Karl Fogel <kfogel@red-bean.com>, Stefan Monnier <monnier@iro.umontreal.ca>
> 
> > In the library level we could (and maybe should) implement both (OT and
> > CRDT). This will only require a hint from the server side (or the
> > connection starter client) to inform the client's libraries "what to use
> > this time". Then practice will say what's better.
> > 
> > ATM we could start with a C library in order to provide a nice free
> > back-end editor agnostic. Because IMO the most interesting part of this
> > will be to open emacs to interact with other editors with a free
> > infrastructure emacs-centric.
> 
> I’m not sure if it’s easy (or worth the effort) to provide CRDT in an 
> editor-agnostic way from C-level. Seems that the most natural
> choice is to tag characters with CRDT data structures. Emacs has
> text property which is very suitable for this, and I’m implementing
> it in this way as an Elisp library. However, this relies on the particular
> data structure for buffer/text emacs uses.
> 
> If implemented as a separate C library, I imagine the CRDT library
> need to have its own buffer/text data structure and somehow keep
> in sync with Emacs’ .. doesn’t sound so clean.

I'm probably missing something because I don't understand the problem
that worries you.  Given the information about how to update a buffer,
isn't it relatively easy to generate a series of insert/delete/replace
operations from that information?  And if so, those actions can be
almost trivially converted to Emacs primitives that manipulate buffer
text.  What am I missing?

Re: Question collaborative editing.

[Eli Zaretskii]

> Date: Wed, 30 Sep 2020 02:13:33 +0200
> From: Ergus <spacibba@aol.com>
> Cc: Karl Fogel <kfogel@red-bean.com>,
>  Emacs developers <emacs-devel@gnu.org>, Fermin <fmfs@posteo.net>,
>  Noam Postavsky <npostavs@gmail.com>, Jean Louis <bugs@gnu.support>,
>  Stefan Monnier <monnier@iro.umontreal.ca>
> 
> I would be concerned about the performance (but I am paranoid about
> Emacs performance anyway so don't worry for that). AFAIR emacs buffers
> are stored in a contiguous way (I may be wrong) and in that case the
> search with CRDT may be expensive. At least as it was explained here (in
> the optimization section):

I'm not sure I understand why you are bothered by the low-level
details of buffer text implementation.  You shouldn't need to
manipulate it at a level lower than calling the functions implemented
in insdel.c.  And as I wrote elsewhere in this thread, I don't yet
understand why Lisp primitives like 'insert' and 'delete-region'
aren't good enough.

Re: Question collaborative editing.

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 2083 bytes --]

>> I’m not sure if it’s easy (or worth the effort) to provide CRDT in an 
>> editor-agnostic way from C-level. Seems that the most natural
>> choice is to tag characters with CRDT data structures. Emacs has
>> text property which is very suitable for this, and I’m implementing
>> it in this way as an Elisp library. However, this relies on the particular
>> data structure for buffer/text emacs uses.
>> 
>> If implemented as a separate C library, I imagine the CRDT library
>> need to have its own buffer/text data structure and somehow keep
>> in sync with Emacs’ .. doesn’t sound so clean.
> 
> I'm probably missing something because I don't understand the problem
> that worries you.  Given the information about how to update a buffer,
> isn't it relatively easy to generate a series of insert/delete/replace
> operations from that information?  And if so, those actions can be
> almost trivially converted to Emacs primitives that manipulate buffer
> text.  What am I missing?
I was just thinking in general, keeping several copy of data and trying 
to get in sync is a likely source of bug and maintenance problems. 
Suppose there are some bug in the implementation and the CRDT lose
track of some part of the text. If CRDT is attached to Emacs buffer, all
operation will still operate almost correctly and resolve to the
correct intents — because they are item based rather than buffer
position based. Suppose it’s implemented in C library way and C
library communicate to Emacs via buffer position information —
the extra text will make the position off and completely mess up
any further operation on this buffer.

Also this scenario doesn’t necessarily comes from a bug in CRDT
implementation — there’s inhibit-modification-hooks which may
prevent some changes to be told to the C library.

I agree that there’s no problem with C library if everything behaves
well — I don’t like the idea of keeping two replicate of data to keep
in sync (the whole CRDT is to solve this) but it might be my personal
opinion.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing.

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> Date: Wed, 30 Sep 2020 15:47:04 +0000
> Cc: Ergus <spacibba@aol.com>, Fermin <fmfs@posteo.net>,
>  Jean Louis <bugs@gnu.support>, Noam Postavsky <npostavs@gmail.com>,
>  Emacs developers <emacs-devel@gnu.org>, Karl Fogel <kfogel@red-bean.com>,
>  Stefan Monnier <monnier@iro.umontreal.ca>
> 
> 
> [1:text/plain Hide]
> 
> >> I’m not sure if it’s easy (or worth the effort) to provide CRDT in an 
> >> editor-agnostic way from C-level. Seems that the most natural
> >> choice is to tag characters with CRDT data structures. Emacs has
> >> text property which is very suitable for this, and I’m implementing
> >> it in this way as an Elisp library. However, this relies on the particular
> >> data structure for buffer/text emacs uses.
> >> 
> >> If implemented as a separate C library, I imagine the CRDT library
> >> need to have its own buffer/text data structure and somehow keep
> >> in sync with Emacs’ .. doesn’t sound so clean.
> > 
> > I'm probably missing something because I don't understand the problem
> > that worries you.  Given the information about how to update a buffer,
> > isn't it relatively easy to generate a series of insert/delete/replace
> > operations from that information?  And if so, those actions can be
> > almost trivially converted to Emacs primitives that manipulate buffer
> > text.  What am I missing?
> I was just thinking in general, keeping several copy of data and trying 
> to get in sync is a likely source of bug and maintenance problems. 
> Suppose there are some bug in the implementation and the CRDT lose
> track of some part of the text. If CRDT is attached to Emacs buffer, all
> operation will still operate almost correctly and resolve to the
> correct intents — because they are item based rather than buffer
> position based. Suppose it’s implemented in C library way and C
> library communicate to Emacs via buffer position information —
> the extra text will make the position off and completely mess up
> any further operation on this buffer.

That's the problem collaborative editing needs to solve, isn't it?
The communications between clients should allow resolution of
differences in buffer position, and detection of losing track of text
modifications.

I don't yet see how the need to solve this problems would require
working on the lowest level of the buffer-text implementation.

> Also this scenario doesn’t necessarily comes from a bug in CRDT
> implementation — there’s inhibit-modification-hooks which may
> prevent some changes to be told to the C library.

You don't need buffer-modification hooks, you can use other existing
mechanisms.  Emacs knows that buffer text was changed even if the
modification hooks were inhibited.

> I agree that there’s no problem with C library if everything behaves
> well — I don’t like the idea of keeping two replicate of data to keep
> in sync (the whole CRDT is to solve this) but it might be my personal
> opinion.

I don't think I understand what you mean by "C library" here.

Re: Question collaborative editing.

[Karl Fogel]

On 30 Sep 2020, Ergus wrote:
>On Wed, Sep 30, 2020 at 02:40:26AM -0500, Karl Fogel wrote:
>>On 30 Sep 2020, Ergus wrote:
>>>I would be concerned about the performance (but I am paranoid about
>>>Emacs performance anyway so don't worry for that). AFAIR emacs buffers
>>>are stored in a contiguous way (I may be wrong)
>>
>>Information about how Emacs buffers are represented under the hood can be found here:
>>
>>* https://git.savannah.gnu.org/cgit/emacs.git/tree/src/buffer.h#n213
>>
>>* https://www.gnu.org/software/emacs/manual/html_node/elisp/Buffer-Gap.html
>>
>This is interesting indeed.
>
>I was looking at this to compare what's around:
>
>https://ecc-comp.blogspot.com/2015/05/a-brief-glance-at-how-5-text-editors.html

When I responded, I should have added what Eli later said in the thread, though:

There's no reason why the underlying (C level) implementation of buffers should matter here.  I expect that collaborative editing can be successfully implemented entirely in Elisp; it would be very, very surprising if you need to look at or even understand anything at the C level for this.

Best regards,
-Karl

Re: Question collaborative editing.

[Qiantan Hong]

[-- Attachment #1.1: Type: text/plain, Size: 1558 bytes --]

> That's the problem collaborative editing needs to solve, isn't it?
> The communications between clients should allow resolution of
> differences in buffer position, and detection of losing track of text
> modifications.

> I don't think I understand what you mean by "C library" here.

As discussed above, if we implement CRDT as a “editor agnostic
C library”, then there will likely to be an extra synchronization
between C library and emacs.

> The communications between clients should allow resolution of
> differences in buffer position, and detection of losing track of text
> modifications.

I’m talking about synchronization between the buffer data structure
in C library, and buffer data structure in Emacs.
Apparently we’re not going to implement CRDT between them
(otherwise why not just let Emacs powered CRDT talk to each other),
and most likely they will exchange message about raw 
(buffer position, operation, text). But this is the most naive
synchronization mechanism and have no robustness.

> You don't need buffer-modification hooks, you can use other existing
> mechanisms.  Emacs knows that buffer text was changed even if the
> modification hooks were inhibited.
That’s a workaround, but I’m not sure if it can capture user intent
(i.e. it gives operation-wise information rather than just a diff between
versions of buffer). If it can’t then we will need to implement both
update based on modification-hooks and change state of the buffer,
which is an additional impl/maintenance burden.

[-- Attachment #1.2: Type: text/html, Size: 5794 bytes --]

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing.

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 857 bytes --]


>> You don't need buffer-modification hooks, you can use other existing
>> mechanisms.  Emacs knows that buffer text was changed even if the
>> modification hooks were inhibited.
> That’s a workaround, but I’m not sure if it can capture user intent
> (i.e. it gives operation-wise information rather than just a diff between
> versions of buffer). If it can’t then we will need to implement both
> update based on modification-hooks and change state of the buffer,
> which is an additional impl/maintenance burden.
I see the difference as, if implemented as a C library that keep a buffer
and synchronize with emacs, then this synchronization *must* be
absolutely correct, otherwise it mess up any operation on the buffer.
However if the CRDT is kept on the emacs buffer itself, it’s much
more tolerant to untracked text in the buffer.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing.

[Stefan Monnier]

> That’s a workaround, but I’m not sure if it can capture user intent
> (i.e. it gives operation-wise information rather than just a diff between
> versions of buffer). If it can’t then we will need to implement both
> update based on modification-hooks and change state of the buffer,
> which is an additional impl/maintenance burden.

I think you're worrying about a non-problem: if the modification hooks
aren't run when you need them, then consider it as a bug in the code
that performed those modifications or in the code that prevented the
hooks from being run.


        Stefan

Re: Question collaborative editing.

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> CC: Ergus <spacibba@aol.com>, Fermin <fmfs@posteo.net>,
>         Jean Louis
> 	<bugs@gnu.support>, Noam Postavsky <npostavs@gmail.com>,
>         Emacs developers
> 	<emacs-devel@gnu.org>,
>         Karl Fogel <kfogel@red-bean.com>,
>         Stefan Monnier
> 	<monnier@iro.umontreal.ca>
> Date: Wed, 30 Sep 2020 17:04:09 +0000
> 
> >> You don't need buffer-modification hooks, you can use other existing
> >> mechanisms.  Emacs knows that buffer text was changed even if the
> >> modification hooks were inhibited.
> > That’s a workaround, but I’m not sure if it can capture user intent
> > (i.e. it gives operation-wise information rather than just a diff between
> > versions of buffer). If it can’t then we will need to implement both
> > update based on modification-hooks and change state of the buffer,
> > which is an additional impl/maintenance burden.
> I see the difference as, if implemented as a C library that keep a buffer
> and synchronize with emacs, then this synchronization *must* be
> absolutely correct, otherwise it mess up any operation on the buffer.
> However if the CRDT is kept on the emacs buffer itself, it’s much
> more tolerant to untracked text in the buffer.

I'm sorry, I probably miss too much background information here,
because I don't really understand what is the issue you are describing
here.  E.g., why is user intent important, when all you need is to
keep buffer text synchronized between several sessions?

Re: Question collaborative editing.

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 1366 bytes --]

> I'm sorry, I probably miss too much background information here,
> because I don't really understand what is the issue you are describing
> here.  E.g., why is user intent important, when all you need is to
> keep buffer text synchronized between several sessions?

That’s the baseline for collaborative editing, there’s apparently
solution that satisfies this single property (convergence property)
but are extremely bad — e.g. the effect of some user input takes
no effect, or a trivial program displaying an empty buffer for 
every user. Actually if we only want this property there’s much
simpler way than CRDT/OT, e.g. let the server (or one client)
send the whole buffer periodically.

One example of diff not preserving user intent is:
suppose in the buffer there is
“one two one”
And users agree that this sentence should always end in “one”.
Now user A deleted the first two words “one two “
user B inserted “three” between the first “one” and “two”
However, the diff have no idea about users’ agreement
and what exactly user does (unless we get it from modification
hooks). For user A, the diff just see “one” after and assume
that A delete the suffix “two one”. The CRDT then does it job
correctly and gives final synchronization result “one three”.
Clearly, user intent is violated.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing.

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> CC: Ergus <spacibba@aol.com>, Fermin <fmfs@posteo.net>,
>         Jean Louis
> 	<bugs@gnu.support>, Noam Postavsky <npostavs@gmail.com>,
>         Emacs developers
> 	<emacs-devel@gnu.org>,
>         Karl Fogel <kfogel@red-bean.com>,
>         Stefan Monnier
> 	<monnier@iro.umontreal.ca>
> Date: Wed, 30 Sep 2020 17:48:13 +0000
> 
> > I'm sorry, I probably miss too much background information here,
> > because I don't really understand what is the issue you are describing
> > here.  E.g., why is user intent important, when all you need is to
> > keep buffer text synchronized between several sessions?
> 
> That’s the baseline for collaborative editing, there’s apparently
> solution that satisfies this single property (convergence property)
> but are extremely bad — e.g. the effect of some user input takes
> no effect, or a trivial program displaying an empty buffer for 
> every user. Actually if we only want this property there’s much
> simpler way than CRDT/OT, e.g. let the server (or one client)
> send the whole buffer periodically.
> 
> One example of diff not preserving user intent is:
> suppose in the buffer there is
> “one two one”
> And users agree that this sentence should always end in “one”.
> Now user A deleted the first two words “one two “
> user B inserted “three” between the first “one” and “two”
> However, the diff have no idea about users’ agreement
> and what exactly user does (unless we get it from modification
> hooks). For user A, the diff just see “one” after and assume
> that A delete the suffix “two one”. The CRDT then does it job
> correctly and gives final synchronization result “one three”.
> Clearly, user intent is violated.

I understand all that, but (a) you can overcome that by techniques
different from CRDT, and (b) unless we mean very different things by
"user intent", the intent is not important here.  What is important,
AFAIU, is to specify the changes in a way that will produce the
desired result even if the buffer in the other Emacs was meanwhile
modified.

Re: Question collaborative editing.

[Ergus]

On Wed, Sep 30, 2020 at 09:08:27PM +0300, Eli Zaretskii wrote:
>> From: Qiantan Hong <qhong@mit.edu>
>> CC: Ergus <spacibba@aol.com>, Fermin <fmfs@posteo.net>,
>>         Jean Louis
>> 	<bugs@gnu.support>, Noam Postavsky <npostavs@gmail.com>,
>>         Emacs developers
>> 	<emacs-devel@gnu.org>,
>>         Karl Fogel <kfogel@red-bean.com>,
>>         Stefan Monnier
>> 	<monnier@iro.umontreal.ca>
>> Date: Wed, 30 Sep 2020 17:48:13 +0000
>>
>> > I'm sorry, I probably miss too much background information here,
>> > because I don't really understand what is the issue you are describing
>> > here.  E.g., why is user intent important, when all you need is to
>> > keep buffer text synchronized between several sessions?
>>
>> That’s the baseline for collaborative editing, there’s apparently
>> solution that satisfies this single property (convergence property)
>> but are extremely bad — e.g. the effect of some user input takes
>> no effect, or a trivial program displaying an empty buffer for
>> every user. Actually if we only want this property there’s much
>> simpler way than CRDT/OT, e.g. let the server (or one client)
>> send the whole buffer periodically.
>>
>> One example of diff not preserving user intent is:
>> suppose in the buffer there is
>> “one two one”
>> And users agree that this sentence should always end in “one”.
>> Now user A deleted the first two words “one two “
>> user B inserted “three” between the first “one” and “two”
>> However, the diff have no idea about users’ agreement
>> and what exactly user does (unless we get it from modification
>> hooks). For user A, the diff just see “one” after and assume
>> that A delete the suffix “two one”. The CRDT then does it job
>> correctly and gives final synchronization result “one three”.
>> Clearly, user intent is violated.
>
>I understand all that, but (a) you can overcome that by techniques
>different from CRDT, and (b) unless we mean very different things by
>"user intent", the intent is not important here.  What is important,
>AFAIU, is to specify the changes in a way that will produce the
>desired result even if the buffer in the other Emacs was meanwhile
>modified.

Hi Eli:

I will probably describe the obvious, but it seems that there is a miss
communication here.

The CRDT algorithm works in a way that every single character in the
buffer has a unique ID if you have "ab" then a could be zero and b could
be 1... Inserting a char between them (acb) will make c be 0.5; adding
another (acdb) will make d=0.75 (this is just a simplified example)

So adding a letter only needs to contain the new number and the char to
insert {0.5; c} {0.75; d}. And to insert "c" we only need to find which
char in the buffer are x < 0.5 and x+1 > 0.5.

If I understood the Qiantan idea; his approach is to add such ID as a
text property within the emacs buffer. To modify the buffer it is only
needed to search (go to) the property ID of text chars/words whatever in
the emacs buffer and perform the action. So the error probability is
smaller because the IDs ARE in the text itself.

This will reduce undetected errors inserting in the wrong positions when
translating form the CRDT ID to the real "global" buffer position to do
an "insert". Which could happen in some corner cases... (lets say
basically synchronization errors between local modifications (which
modify local absolute positions) and processing remote ones using
outdated information (translating to global indices)...

Said that; the external library approach seemed to work with tandem
https://github.com/jscheid/tandem-emacs

I am not sire how well it did.

My previous concerns about the internal storage and performance was
because the linear search of text properties in a long linear buffer may
be very expensive as described in the refereed paper; so using a two
steps search improved performance as described there... There are also
other optimizations like grouping operations or remove redundant ones
that with the C approach could reduce the operation that arrive to
emacs.  Probably in our case the improvement may be to use a bisection
search and position hints.

Any way. I am pretty sure that the Qiantan lisp only implementation
could work; but performance is something that needs to be tested
(hopefully performance issues will be negligible compared to network
latency).

The C library implementation was my utopic ideal implementation in order
to make it efficient, portable and reusable with other editors... But it
will somehow require to start a completely independent project from
scratch and would require much more time to implement (algorithm,
network handling, json, synchronization between clients and with
emacs)... And basically emacs provides all that. Maybe this time simple
is not better, but more realistic.

Re: Question collaborative editing.

[Eli Zaretskii]

> Date: Thu, 1 Oct 2020 01:11:59 +0200
> From: Ergus <spacibba@aol.com>
> Cc: Qiantan Hong <qhong@mit.edu>, fmfs@posteo.net, bugs@gnu.support,
> 	npostavs@gmail.com, emacs-devel@gnu.org, kfogel@red-bean.com,
> 	monnier@iro.umontreal.ca
> 
> If I understood the Qiantan idea; his approach is to add such ID as a
> text property within the emacs buffer. To modify the buffer it is only
> needed to search (go to) the property ID of text chars/words whatever in
> the emacs buffer and perform the action. So the error probability is
> smaller because the IDs ARE in the text itself.

IMO, that is not a good idea, to put it mildly.  Since these
properties can potentially span very few characters, or even change
the value at each buffer position in extreme cases, they will most
probably slow down redisplay, perhaps even significantly so.  The
problem here is that the display engine always knows where's the next
buffer position, called "stop position" at which text properties
change values.  Between the stop positions, the display engine runs at
full throttle, using the information about properties (notably, faces)
and overlays it computed at the last stop position.  But when it comes
to the next stop position, it stops and reconsiders all the stuff that
can affect display: text properties (faces, invisibility), overlays,
composed characters, etc., something that requires non-trivial
processing.  Having text properties change too often will cause
slowdown, even though these properties don't affect redisplay.

You can measure this slowdown by putting some text property on each
character in a buffer (the property should have a different value for
each character), then timing redisplay in a large enough buffer with
such properties.  For example, lean on the DOWN arrow and see how long
it takes to scroll through a large file; or run one of the scroll-down
benchmarks that were posted here in the past.

Maybe I'm being overly pessimistic, but I expect slower redisplay with
so many text property changes.  So I definitely wouldn't recommend
going that way.

> This will reduce undetected errors inserting in the wrong positions when
> translating form the CRDT ID to the real "global" buffer position to do
> an "insert". Which could happen in some corner cases... (lets say
> basically synchronization errors between local modifications (which
> modify local absolute positions) and processing remote ones using
> outdated information (translating to global indices)...

I'm not sure we need to implement this in Emacs.  For example, Tandem
doesn't require this from its plugins; presumably, it builds the
character IDs internally?  You could look at its implementation of
CRDT and take the ideas from there; AFAIU, it requires only a couple
of very simple operations to be implemented by plugins.

I also am not sure we should disregard the OT-based designs.  It is
true that they scale worse than CRDTs, but do we really envision
groups of tens, let alone hundreds or thousands of users working at
the same time in Emacs on the same document?  For smaller groups,
OT-based design might be good enough, and if it is simpler to
implement and requires less from Emacs core, maybe this possibility
should also be considered?

> My previous concerns about the internal storage and performance was
> because the linear search of text properties in a long linear buffer may
> be very expensive as described in the refereed paper;

Text property search in Emacs is not linear, it uses interval trees.
The problem is not in the search, the problem is in how often we will
need to do this when we traverse the buffer text, as redisplay does.

Re: Question collaborative editing.

[Jean Louis]

* Eli Zaretskii <eliz@gnu.org> [2020-09-30 19:20]:
> That's the problem collaborative editing needs to solve, isn't it?
> The communications between clients should allow resolution of
> differences in buffer position, and detection of losing track of text
> modifications.

Maybe it could be done with some kind of a queue server on TCP port
between the local Emacs and clients.

Re: Question collaborative editing.

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 2782 bytes --]


> IMO, that is not a good idea, to put it mildly.  Since these
> properties can potentially span very few characters, or even change
> the value at each buffer position in extreme cases, they will most
> probably slow down redisplay, perhaps even significantly so.  The
> problem here is that the display engine always knows where's the next
> buffer position, called "stop position" at which text properties
> change values.  Between the stop positions, the display engine runs at
> full throttle, using the information about properties (notably, faces)
> and overlays it computed at the last stop position.  But when it comes
> to the next stop position, it stops and reconsiders all the stuff that
> can affect display: text properties (faces, invisibility), overlays,
> composed characters, etc., something that requires non-trivial
> processing.  Having text properties change too often will cause
> slowdown, even though these properties don't affect redisplay.

Thanks for the insight, I’ve never knew Emacs internal so much
to be able to consider this. The optimistic side is that I’m using
the merge-item approach as in Yjs so hopefully text property change
is not so frequent…

> I'm not sure we need to implement this in Emacs.  For example, Tandem
> doesn't require this from its plugins; presumably, it builds the
> character IDs internally?  You could look at its implementation of
> CRDT and take the ideas from there; AFAIU, it requires only a couple
> of very simple operations to be implemented by plugins.

IMO the problem of this approach is that if there’s any desync between
buffer in the library and buffer in the editor the outcome is disastrous,
and this can not only come from some bug — Emacs and C library
has to talk asynchronously and bidirectionally, which means theoretically
there will be case that Emacs does some change, sent a message to
C library, but some message before C library receive the change from
Emacs arrives later 
    Emacs C-Library
 |      |  \   /  |
 |      |    X   |
 |      |  /   \  |
V
Although the latency window on local machine is very small, the latency
of Emacs Lisp interpreter responding to events is a another complication...
There are way to work around this of course, but it’ll definitely require more
work rather than just sending simple operation and buffer position, at least
there need to be UID for each operation and some transaction mechanism.

I see that Tandem send old text and new text with the text operation. This
is definitely a workaround with no correctness guarantee, and it doesn’t
specify what to do when a desync happens. But those are just my mutering
about correctness, I can imagine that it might work good enough in practice.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing.

[Ergus]

On Thu, Oct 01, 2020 at 04:40:59PM +0300, Eli Zaretskii wrote:
>> Date: Thu, 1 Oct 2020 01:11:59 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: Qiantan Hong <qhong@mit.edu>, fmfs@posteo.net, bugs@gnu.support,
>> 	npostavs@gmail.com, emacs-devel@gnu.org, kfogel@red-bean.com,
>> 	monnier@iro.umontreal.ca
>>
>> If I understood the Qiantan idea; his approach is to add such ID as a
>> text property within the emacs buffer. To modify the buffer it is only
>> needed to search (go to) the property ID of text chars/words whatever in
>> the emacs buffer and perform the action. So the error probability is
>> smaller because the IDs ARE in the text itself.
>
>IMO, that is not a good idea, to put it mildly.  Since these
>properties can potentially span very few characters, or even change
>the value at each buffer position in extreme cases, they will most
>probably slow down redisplay, perhaps even significantly so.  The
>problem here is that the display engine always knows where's the next
>buffer position, called "stop position" at which text properties
>change values.  Between the stop positions, the display engine runs at
>full throttle, using the information about properties (notably, faces)
>and overlays it computed at the last stop position.  But when it comes
>to the next stop position, it stops and reconsiders all the stuff that
>can affect display: text properties (faces, invisibility), overlays,
>composed characters, etc., something that requires non-trivial
>processing.  Having text properties change too often will cause
>slowdown, even though these properties don't affect redisplay.
>
>You can measure this slowdown by putting some text property on each
>character in a buffer (the property should have a different value for
>each character), then timing redisplay in a large enough buffer with
>such properties.  For example, lean on the DOWN arrow and see how long
>it takes to scroll through a large file; or run one of the scroll-down
>benchmarks that were posted here in the past.
>
>Maybe I'm being overly pessimistic, but I expect slower redisplay with
>so many text property changes.  So I definitely wouldn't recommend
>going that way.
>
Oh; you are right. Now that I remember some of that code in the display
engine you are totally right.

>> This will reduce undetected errors inserting in the wrong positions when
>> translating form the CRDT ID to the real "global" buffer position to do
>> an "insert". Which could happen in some corner cases... (lets say
>> basically synchronization errors between local modifications (which
>> modify local absolute positions) and processing remote ones using
>> outdated information (translating to global indices)...
>
>I'm not sure we need to implement this in Emacs.  For example, Tandem
>doesn't require this from its plugins; presumably, it builds the
>character IDs internally?  You could look at its implementation of
>CRDT and take the ideas from there; AFAIU, it requires only a couple
>of very simple operations to be implemented by plugins.
>
Indeed; that's why I was considering the external library in C. In
general it is actually feasible. Sadly the translation from Tandem to C
will require some time because it is split with many code mixtures and
non of them in C... but if you think is better we could do that (with
SOME time).

Any way we could wait for the Qiantan implementation and consider some
optimizations if you think it worth the effort.

>I also am not sure we should disregard the OT-based designs.  It is
>true that they scale worse than CRDTs, but do we really envision
>groups of tens, let alone hundreds or thousands of users working at
>the same time in Emacs on the same document?  For smaller groups,
>OT-based design might be good enough, and if it is simpler to
>implement and requires less from Emacs core, maybe this possibility
>should also be considered?
>
Indeed; If we use an external library; one of the advantages is that we
could have both because for the emacs side will be exactly the same. It
only received a "list" of changes to implement then a set of external
ones. And the network implementation and so on will be more or less the
same. IMO the hardest part of this is all the network managing, server
and so on... not CRDT is OT implementations.

>> My previous concerns about the internal storage and performance was
>> because the linear search of text properties in a long linear buffer may
>> be very expensive as described in the refereed paper;
>
>Text property search in Emacs is not linear, it uses interval trees.
>The problem is not in the search, the problem is in how often we will
>need to do this when we traverse the buffer text, as redisplay does.

Re: Question collaborative editing.

[Ergus]

On Thu, Oct 01, 2020 at 05:11:44PM +0300, Jean Louis wrote:
>* Eli Zaretskii <eliz@gnu.org> [2020-09-30 19:20]:
>> That's the problem collaborative editing needs to solve, isn't it?
>> The communications between clients should allow resolution of
>> differences in buffer position, and detection of losing track of text
>> modifications.
>
>Maybe it could be done with some kind of a queue server on TCP port
>between the local Emacs and clients.

It is a bit more complex, because for remote editing it is not easy to
establish connection in different NAP, cross firewalls, and so on. And
as I said before, we don't have resources to provide a server service
for this that runs "our" CRTD server.

Withing the same network it still requires a sort of local service to
solve the conflicts. And that also needs to be implemented...

Re: Question collaborative editing.

[Pankaj Jangid]

On Thu, Oct 01 2020, Ergus wrote:

> It is a bit more complex, because for remote editing it is not easy to
> establish connection in different NAP, cross firewalls, and so on. And
> as I said before, we don't have resources to provide a server service
> for this that runs "our" CRTD server.
>
> Withing the same network it still requires a sort of local service to
> solve the conflicts. And that also needs to be implemented...

Well, we can start with something simple. How about implementing
collaboration on local area network first. I remember we had fun with
'talk'. Once implemented on LAN, users can then figure out how to work
across continents using some kind of tunnel.

An example use-case might be:

user1:
1. C-x C-f (find-file)
2. M-x enable-collaboration ; this generates a session-key
3. Share session key with collaborators via chat/email

user2: (via Tramp)
1. C-x C-f /collab:session-key@ip-address:filename <RET>

Users get cursors with different colours.

The advantage of using Tramp is that users can always use Ad-hoc
multi-hops.

Re: Question collaborative editing.

[Ergus]

On Thu, Oct 01, 2020 at 10:14:34PM +0530, Pankaj Jangid wrote:
>On Thu, Oct 01 2020, Ergus wrote:
>
>> It is a bit more complex, because for remote editing it is not easy to
>> establish connection in different NAP, cross firewalls, and so on. And
>> as I said before, we don't have resources to provide a server service
>> for this that runs "our" CRTD server.
>>
>> Withing the same network it still requires a sort of local service to
>> solve the conflicts. And that also needs to be implemented...
>
>Well, we can start with something simple. How about implementing
>collaboration on local area network first. I remember we had fun with
>'talk'. Once implemented on LAN, users can then figure out how to work
>across continents using some kind of tunnel.
>
>An example use-case might be:
>
>user1:
>1. C-x C-f (find-file)
>2. M-x enable-collaboration ; this generates a session-key
>3. Share session key with collaborators via chat/email
>
>user2: (via Tramp)
>1. C-x C-f /collab:session-key@ip-address:filename <RET>
>
>Users get cursors with different colours.
>
>The advantage of using Tramp is that users can always use Ad-hoc
>multi-hops.
>
Yes, this is exactly my expected workflow more or less. It is p2p as I
have mentioned before I would prefer it to be. The tramp integration is
something way far from my capabilities, but in general that's the
idea ;).

Re: Question collaborative editing.

[Yuri Khan]

On Fri, 2 Oct 2020 at 00:05, Ergus <spacibba@aol.com> wrote:

> >An example use-case might be:
> >
> >user1:
> >1. C-x C-f (find-file)
> >2. M-x enable-collaboration ; this generates a session-key
> >3. Share session key with collaborators via chat/email
> >
> >user2: (via Tramp)
> >1. C-x C-f /collab:session-key@ip-address:filename <RET>
> >
> >Users get cursors with different colours.
> >
> >The advantage of using Tramp is that users can always use Ad-hoc
> >multi-hops.
> >
> Yes, this is exactly my expected workflow more or less. It is p2p as I
> have mentioned before I would prefer it to be. The tramp integration is
> something way far from my capabilities, but in general that's the
> idea ;).

Do not overlook the use case of collaborating on a project containing
multiple files. (I can already see it is a whole other can of worms.)

Re: Question collaborative editing.

[Karl Fogel]

[-- Attachment #1: Type: text/plain, Size: 1490 bytes --]

>IMO the problem of this approach is that if there’s any desync between
>buffer in the library and buffer in the editor the outcome is disastrous,
>and this can not only come from some bug — Emacs and C library
>has to talk asynchronously and bidirectionally, which means theoretically
>there will be case that Emacs does some change, sent a message to
>C library, but some message before C library receive the change from
>Emacs arrives later 
>    Emacs C-Library
> |      |  \   /  |
> |      |    X   |
> |      |  /   \  |
>V
>Although the latency window on local machine is very small, the latency
>of Emacs Lisp interpreter responding to events is a another complication...
>There are way to work around this of course, but it’ll definitely require more
>work rather than just sending simple operation and buffer position, at least
>there need to be UID for each operation and some transaction mechanism.
>
>I see that Tandem send old text and new text with the text operation. This
>is definitely a workaround with no correctness guarantee, and it doesn’t
>specify what to do when a desync happens. But those are just my mutering
>about correctness, I can imagine that it might work good enough in practice.

Well, hey, Emacs has an `md5' function :-).  For example, the result of

  (insert (md5 (current-buffer)))

is

  5d260ed0f9899cbb9fc53ec96324f5d5

(Of course, the result was wrong as soon as I evaluated it!)

Best regards,
-Karl

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

Re: Question collaborative editing.

[Pankaj Jangid]

On Fri, Oct 02 2020, Yuri Khan wrote:

>> >An example use-case might be:
>> >
>> >user1:
>> >1. C-x C-f (find-file)
>> >2. M-x enable-collaboration ; this generates a session-key
>> >3. Share session key with collaborators via chat/email
>> >
>> >user2: (via Tramp)
>> >1. C-x C-f /collab:session-key@ip-address:filename <RET>
>> >
>> >Users get cursors with different colours.
>> >
>> >The advantage of using Tramp is that users can always use Ad-hoc
>> >multi-hops.
>> >
>> Yes, this is exactly my expected workflow more or less. It is p2p as I
>> have mentioned before I would prefer it to be. The tramp integration is
>> something way far from my capabilities, but in general that's the
>> idea ;).
>
> Do not overlook the use case of collaborating on a project containing
> multiple files. (I can already see it is a whole other can of worms.)

I thought about this while writing the above use-case. Hence used the
generic term /session-key/. A session can use the project infrastructure
to list and expose files. See the last part in the above Tramp
call. Filename can be specified there.

Re: Question collaborative editing.

[Michael Albinus]

Ergus <spacibba@aol.com> writes:

>>> It is a bit more complex, because for remote editing it is not easy to
>>> establish connection in different NAP, cross firewalls, and so on. And
>>> as I said before, we don't have resources to provide a server service
>>> for this that runs "our" CRTD server.
>>>
>>> Withing the same network it still requires a sort of local service to
>>> solve the conflicts. And that also needs to be implemented...
>>
>>Well, we can start with something simple. How about implementing
>>collaboration on local area network first. I remember we had fun with
>>'talk'. Once implemented on LAN, users can then figure out how to work
>>across continents using some kind of tunnel.
>>
>>An example use-case might be:
>>
>>user1:
>>1. C-x C-f (find-file)
>>2. M-x enable-collaboration ; this generates a session-key
>>3. Share session key with collaborators via chat/email
>>
>>user2: (via Tramp)
>>1. C-x C-f /collab:session-key@ip-address:filename <RET>
>>
>>Users get cursors with different colours.
>>
>>The advantage of using Tramp is that users can always use Ad-hoc
>>multi-hops.
>>
> Yes, this is exactly my expected workflow more or less. It is p2p as I
> have mentioned before I would prefer it to be. The tramp integration is
> something way far from my capabilities, but in general that's the
> idea ;).

I don't want to be the grinch, but pls remember what Tramp is: a library
for alternative implementations of file operations. I don't see yet how
/collab:session-key@ip-address:filename fits into the game, but maybe it
needs more details to understand. For example, which file operations
shall be treated.

Furthermore, I also don't understand what needs to be done on the remote
side wrt collaboration. A remote file in Emacs is still a file,
connected to a buffer. All primitive file operations can be applied on
this.

Best regards, Michael.

Re: Question collaborative editing.

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Jean Louis <bugs@gnu.support>,  Eli Zaretskii <eliz@gnu.org>,  Qiantan
>  Hong <qhong@mit.edu>,  fmfs@posteo.net,  npostavs@gmail.com,
>   emacs-devel@gnu.org,  kfogel@red-bean.com,  monnier@iro.umontreal.ca
> Date: Fri, 02 Oct 2020 09:30:53 +0200
> 
> I don't want to be the grinch, but pls remember what Tramp is: a library
> for alternative implementations of file operations. I don't see yet how
> /collab:session-key@ip-address:filename fits into the game, but maybe it
> needs more details to understand. For example, which file operations
> shall be treated.
> 
> Furthermore, I also don't understand what needs to be done on the remote
> side wrt collaboration. A remote file in Emacs is still a file,
> connected to a buffer. All primitive file operations can be applied on
> this.

I think Michael is right: Tramp and collaborative editing are
orthogonal features.  Collaborative editing is about special handling
of changes to buffer text, and Tramp has nothing to say about that,
AFAIK.

Re: Question collaborative editing.

[Pankaj Jangid]

On Fri, Oct 02 2020, Michael Albinus wrote:

>>>An example use-case might be:
>>>
>>>user1:
>>>1. C-x C-f (find-file)
>>>2. M-x enable-collaboration ; this generates a session-key
>>>3. Share session key with collaborators via chat/email
>>>
>>>user2: (via Tramp)
>>>1. C-x C-f /collab:session-key@ip-address:filename <RET>
>>>
>>>Users get cursors with different colours.
>>>
>>>The advantage of using Tramp is that users can always use Ad-hoc
>>>multi-hops.
>>>
>> Yes, this is exactly my expected workflow more or less. It is p2p as I
>> have mentioned before I would prefer it to be. The tramp integration is
>> something way far from my capabilities, but in general that's the
>> idea ;).
>
> I don't want to be the grinch, but pls remember what Tramp is: a library
> for alternative implementations of file operations. I don't see yet how
> /collab:session-key@ip-address:filename fits into the game, but maybe it
> needs more details to understand. For example, which file operations
> shall be treated.
>
> Furthermore, I also don't understand what needs to be done on the remote
> side wrt collaboration. A remote file in Emacs is still a file,
> connected to a buffer. All primitive file operations can be applied on
> this.

Tramp example was basically to consider the point that people are
already familier with the syntax to open a remote file (format -
/protocol:identification:resource). *We can propose and use any other
familier format*.

Secondly, I was not thinking about the implementation. I was thinking
from a user's perspective. I was thinking of these two properties - (1)
Attaches a remote file with a buffer (2) If the user is using automatic
desktop-save and desktop-read, then tramp buffers are not reconnected.

Re: Question collaborative editing.

[Ergus]

On Fri, Oct 02, 2020 at 09:30:53AM +0200, Michael Albinus wrote:
>Ergus <spacibba@aol.com> writes:
>
>>>> It is a bit more complex, because for remote editing it is not easy to
>>>> establish connection in different NAP, cross firewalls, and so on. And
>>>> as I said before, we don't have resources to provide a server service
>>>> for this that runs "our" CRTD server.
>>>>
>>>> Withing the same network it still requires a sort of local service to
>>>> solve the conflicts. And that also needs to be implemented...
>>>
>>>Well, we can start with something simple. How about implementing
>>>collaboration on local area network first. I remember we had fun with
>>>'talk'. Once implemented on LAN, users can then figure out how to work
>>>across continents using some kind of tunnel.
>>>
>>>An example use-case might be:
>>>
>>>user1:
>>>1. C-x C-f (find-file)
>>>2. M-x enable-collaboration ; this generates a session-key
>>>3. Share session key with collaborators via chat/email
>>>
>>>user2: (via Tramp)
>>>1. C-x C-f /collab:session-key@ip-address:filename <RET>
>>>
>>>Users get cursors with different colours.
>>>
>>>The advantage of using Tramp is that users can always use Ad-hoc
>>>multi-hops.
>>>
>> Yes, this is exactly my expected workflow more or less. It is p2p as I
>> have mentioned before I would prefer it to be. The tramp integration is
>> something way far from my capabilities, but in general that's the
>> idea ;).
>
>I don't want to be the grinch, but pls remember what Tramp is: a library
>for alternative implementations of file operations. I don't see yet how
>/collab:session-key@ip-address:filename fits into the game, but maybe it
>needs more details to understand. For example, which file operations
>shall be treated.
>
Hi Michael:

I don't have any clear idea yet (and don;t know the TRAMP code). But, as
this will be implemented more or less as a server-client, the only thing
tramp would need to do is to connect (from the client side), identify
and request the file... very similar to opening an ssh file, but maling
a local copy and initializing the server CRDT/OT local service as
client. The rest of the process will be handled internally by the
collaborative api like any other file from the Tramp's point of view.

I think tramp is the preferred place to avoid creating another interface
for the user (extra names, commands and so on). But actually the only
part that Tramp should take care may be the initial "login" and probably
the logout/close-connection.

In any case this could also be implemented from outside tramp and add a
condition to commands like find-file... But IMO these are just
details.. we need to make it work first.

>Furthermore, I also don't understand what needs to be done on the remote
>side wrt collaboration. A remote file in Emacs is still a file,
>connected to a buffer. All primitive file operations can be applied on
>this.
>
In local or remote side the requirements are similar (From the tramp
point of view):

Server side:
Start the OT/CRDT server/service
Associate with a local file(s)

Client side:
Start the OT/CRDT server/service (as client)
Connect to remote server
Identify
Request/create remote file copy to work locally.
Associate

>Best regards, Michael.
>

Best,
Ergus

Re: Question collaborative editing.

[Michael Albinus]

Ergus <spacibba@aol.com> writes:

> Hi Michael:

Hi Ergus,

>>Furthermore, I also don't understand what needs to be done on the remote
>>side wrt collaboration. A remote file in Emacs is still a file,
>>connected to a buffer. All primitive file operations can be applied on
>>this.
>>
> In local or remote side the requirements are similar (From the tramp
> point of view):
>
> Server side:
> Start the OT/CRDT server/service
> Associate with a local file(s)
>
> Client side:
> Start the OT/CRDT server/service (as client)
> Connect to remote server
> Identify
> Request/create remote file copy to work locally.
> Associate

Tramp gives you start-file-process and make-process. This shall be
sufficient to create a client/server connection. What data are over the
line, isn't Tramp's business.

And there's also make-network-process, which doesn't need Tramp at all.

> Best,
> Ergus

Best regards, Michael.

Emacs + Etherpad compatibility? (was: Question collaborative editing.)

[Karl Fogel]

This discussion is exciting.  I regret that I don't have to work on actual implementation, but maybe posting this thought here can be a small contribution:

The fastest route to collaborative editing in Emacs might be to make an Emacs-side client implementation for the protocol used in Etherpad Lite.  

(I'll just call it "Etherpad" from now on, since it's the only actively maintained fork of Etherpad now as far as I know.)

Etherpad is browser-based and is the most widely-used free software collaborative editor.  It's been very successful.  The entities it operates on are already pretty compatible with Emacs -- it's basically plain text with some simple formatting: bold, italic, underlining, strike-through, simple bullet-point or numbered lists, authorship colors, and not much else.  (Even the authorship colors don't necessarily have to be represented as colors; that's just the optional display convention in Etherpad.)

The protocol used by Etherpad is EasySync -- see docs here:

https://github.com/ether/etherpad-lite/blob/develop/doc/easysync/easysync-notes.pdf

https://github.com/ether/etherpad-lite/blob/develop/doc/easysync/easysync-full-description.pdf

(Or just do 'git clone https://github.com/ether/etherpad-lite.git' and look in the 'docs/' subdirectory.)

Emacs can certainly display all the things Etherpad can display.  How hard would it be to implement an Emacs client library to handle the protocol?

Personally, that's where I'd start if I had time to work on this.  It would give us both real-time collaborative editing for Emacs, *and* Emacs users being able to real-time collaborate with non-Emacs users on the same document!  This would be an enormous win for Emacs.

Best regards,
-Karl

Re: Emacs + Etherpad compatibility?

[Stefan Monnier]

> The protocol used by Etherpad is EasySync -- see docs here:
>
> https://github.com/ether/etherpad-lite/blob/develop/doc/easysync/easysync-notes.pdf
>
> https://github.com/ether/etherpad-lite/blob/develop/doc/easysync/easysync-full-description.pdf

These docs don't seem quite sufficient to implement a client, but it
looks like it shouldn't be hard to implement it, indeed.

> Personally, that's where I'd start if I had time to work on this.  It would
> give us both real-time collaborative editing for Emacs, *and* Emacs users
> being able to real-time collaborate with non-Emacs users on the same
> document!  This would be an enormous win for Emacs.

That'd be sweet, yes,


        Stefan

Re: Emacs + Etherpad compatibility?

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The protocol used by Etherpad is EasySync -- see docs here:
>>
>> https://github.com/ether/etherpad-lite/blob/develop/doc/easysync/easysync-notes.pdf
>>
>> https://github.com/ether/etherpad-lite/blob/develop/doc/easysync/easysync-full-description.pdf
>
> These docs don't seem quite sufficient to implement a client, but it
> looks like it shouldn't be hard to implement it, indeed.

IIRC, rudel comes with an etherpad backend. I haven't tried it ever, 'tho.

>         Stefan

Best regards, Michael.

Re: Question collaborative editing - Wikipedia reference

[Jean Louis]

Here is Wikipedia reference on collaborative real time editor:
https://en.wikipedia.org/wiki/Collaborative_real-time_editor

As it was in the Mother of All Demo by Douglas Engelbart, it is pity
that Emacs yet does not support such, especially because Emacs can
support such through its Lisp.

On the other hand, I find that collaborative text editing need not
take place in the same time and that revision systems would be quite
enough for larger groups.

For smaller groups with patience, collaborative text editing can take
place even without Internet, as files can be sent by postal mail to
each other. Don't mind my extreme thinking, yet simple exchange of
file by any means, by any type of networking, is quite fine for
collaborative text editing in small groups.

Myself I cannot find practical uses for real-time collaborative
editing.

Additionally, collaborative editing would require some kind of speech
conversation, as otherwise it becomes foolish to edit
"collaboratively" if collaborators are not really collaborating, they
need to have communication system, and not just text that is edited
remotely. That would mean they are not coordinating with each other,
but doing what they think is best.

Normal editing involves first thinking, thus communication with
oneself, it can be fraction of a second, and then typing.

Collaborative editing should start with thinking of collaborators,
which means communication between collaborators, and then decision to
type. Without communication in background, creation of collaborative
editing does not impress me.

If anybody has practical use from real life, I would like to know
about it.

Re: Question collaborative editing - Wikipedia reference

[Caio Henrique]

Jean Louis <bugs@gnu.support> writes:

> If anybody has practical use from real life, I would like to know
> about it.

Last week a friend and I had to write together a text document for a
university class. We used google meet to talk and google docs to work on
the text document at the same time. That was very efficient and we
finished our work in just a couple of hours. This use case is very
common in my university.   

None of my friends use Emacs (they use libreoffice, notepad, vim, ms word
etc), so an Emacs-to-Emacs only collabaration system would not be useful
for me. 

Re: Question collaborative editing - Wikipedia reference

[Jean Louis]

* Caio Henrique <caiohcs0@gmail.com> [2020-10-04 21:45]:
> Jean Louis <bugs@gnu.support> writes:
> 
> > If anybody has practical use from real life, I would like to know
> > about it.
> 
> Last week a friend and I had to write together a text document for a
> university class. We used google meet to talk and google docs to work on
> the text document at the same time. That was very efficient and we
> finished our work in just a couple of hours. This use case is very
> common in my university.

Thank you for description of the practical use case. 

Alright. It would not work for me. That would mean that I would expose
my information from business to Google servers, and I have no clue of
security.

I have just tried Gobby editor, it works well for collaboration in
real-time, and by launching it creates the server. 

By the way, for talking I would use self-hosted Mumble server, client
works with every device.

> None of my friends use Emacs (they use libreoffice, notepad, vim, ms word
> etc), so an Emacs-to-Emacs only collabaration system would not be useful
> for me.

libreoffice and vim are free software

Who uses notepad, does not need more than notepad.

MsWord is proprietary software, I would never recommend students to
get dependent on such.

Thus good solution for time being would be Gobby and Jitsi or Mumble
speech servers, both are free software.

Re: Question collaborative editing - Wikipedia reference

[Qiantan Hong]

[-- Attachment #1.1: Type: text/plain, Size: 682 bytes --]

I’ve finished the initial work for a CRDT based
collaborative editing Elisp package. The .el is attached.

The focus is on implementing the core algorithm
and evaluating performance, so no cursor indicator,
username display etc was implemented. However,
they should be easy to add and not relevant to performance
concern.

I’ve tested it with my friends and the performance seems 
reasonable. Using LogootSplit algorithm, the number of
text property changes are relatively small.

M-x crdt-test-server starts a local server at port 1333,
and M-x crdt-test-client connects to 127.0.0.1:1333.
Or use crdt-serve-buffer and crdt-connect for arbitrary
address/port.

[-- Attachment #1.2: crdt.el --]
[-- Type: application/octet-stream, Size: 21611 bytes --]

(require 'cl-lib)

;; Loogoot split algorithm
;; André, Luc, et al. "Supporting adaptable granularity of changes for massive-scale collaborative editing." 9th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing. IEEE, 2013.
(defvar crdt--local-clock 0)
(defvar crdt--local-id)
(defvar crdt--inhibit-update nil "When set, don't call CRDT--LOCAL-* on change.
This is useful for functions that apply remote change to local buffer,
to avoid recusive calling of CRDT synchronization functions.")

;; CRDT IDs are represented by unitbyte strings (for efficient comparison)
;; Every two bytes represent a big endian encoded integer
;; For base IDs, last two bytes are always representing site ID
(defconst crdt--max-value (lsh 1 16))
;; (defconst crdt--max-value 4)
;; for debug
(defconst crdt--low-byte-mask 255)
(defsubst crdt--get-two-bytes (string index)
  (logior (lsh (elt string index) 8)
          (elt string (1+ index))))
(defsubst crdt--get-two-bytes-with-offset (string offset index default)
  (cond ((= index (- (string-bytes string) 2))
         offset)
        ((< (1+ index) (string-bytes string))
         (logior (lsh (elt string index) 8)
                 (elt string (1+ index))))
        (t default)))

(defsubst crdt--id-offset (id)
  "Get the literal offset integer from ID.
Note that it might deviate from real offset for a character
in the middle of a block."
  (crdt--get-two-bytes id (- (string-bytes id) 2)))
(defsubst crdt--set-id-offset (id offset)
  (let ((length (string-bytes id)))
    (aset id (- length 2) (lsh offset -8))
    (aset id (- length 1) (logand offset crdt--low-byte-mask))))
(defsubst crdt--id-site (id)
  (crdt--get-two-bytes id (- (string-bytes id) 4)))
(defsubst crdt--generate-id (low-id low-offset high-id high-offset site-id)
  (let* ((l (crdt--get-two-bytes-with-offset low-id low-offset 0 0))
         (h (crdt--get-two-bytes-with-offset high-id high-offset 0 crdt--max-value))
         (bytes (cl-loop for pos from 2 by 2
                         while (< (- h l) 2)
                         append (list (lsh l -8)
                                      (logand l crdt--low-byte-mask))
                         do (setq l (crdt--get-two-bytes-with-offset low-id low-offset pos 0))
                         do (setq h (crdt--get-two-bytes-with-offset high-id high-offset pos crdt--max-value))))
         (m (+ l 1 (random (- h l 1)))))
    (apply #'unibyte-string
           (append bytes (list (lsh m -8)
                               (logand m crdt--low-byte-mask)
                               (lsh site-id -8)
                               (logand site-id crdt--low-byte-mask)
                               0
                               0)))))

;; CRDT-ID text property actually stores a cons of (ID-STRING . END-OF-BLOCK-P)
(defsubst crdt--get-crdt-id-pair (pos &optional obj)
  (get-text-property pos 'crdt-id obj))
(defsubst crdt--get-starting-id (pos &optional obj)
  (car (crdt--get-crdt-id-pair pos obj)))
(defsubst crdt--end-of-block-p (pos &optional obj)
  (cdr (crdt--get-crdt-id-pair pos obj)))
(defsubst crdt--get-starting-id-maybe (pos &optional obj limit)
  (unless (< pos (or limit (point-min)))
    (car (get-text-property pos 'crdt-id obj))))
(defsubst crdt--get-id-offset (starting-id pos &optional obj limit)
  "Get the real offset integer for a character at POS,
assuming the stored literal ID is STARTING-ID."
  (let* ((start-pos (previous-single-property-change (1+ pos) 'crdt-id obj (or limit (point-min)))))
    (+ (- pos start-pos) (crdt--id-offset starting-id))))

(defsubst crdt--set-id (pos id &optional end-of-block obj limit)
  (put-text-property pos (next-single-property-change pos 'crdt-id obj (or limit (point-max))) 'crdt-id (cons id end-of-block) obj))

(defsubst crdt--id-replace-offset (id offset)
  (let ((new-id (substring id)))
    (crdt--set-id-offset new-id offset)
    new-id))
(cl-defmacro crdt--with-insertion-information
    ((beg end &optional beg-obj end-obj beg-limit end-limit) &body body)
  `(let* ((not-begin (> ,beg ,(or beg-limit '(point-min)))) ; if it's nil, we're at the beginning of buffer
          (left-pos (1- ,beg))
          (starting-id-pair (when not-begin (crdt--get-crdt-id-pair left-pos ,beg-obj)))
          (starting-id (if not-begin (car starting-id-pair) ""))
          (left-offset (if not-begin (crdt--get-id-offset starting-id left-pos ,beg-obj ,beg-limit) 0))
          (not-end (< ,end ,(or end-limit '(point-max))))
          (ending-id (if not-end (crdt--get-starting-id ,end ,end-obj) ""))
          (right-offset (if not-end (crdt--id-offset ending-id) 0))
          (beg ,beg)
          (end ,end)
          (beg-obj ,beg-obj)
          (end-obj ,end-obj)
          (beg-limit ,beg-limit)
          (end-limit ,end-limit))
     ,@body))
(defmacro crdt--split-maybe ()
  '(when (and not-end (eq starting-id (crdt--get-starting-id end end-obj)))
     ;; need to split id block
     (crdt--set-id end (crdt--id-replace-offset starting-id (1+ left-offset))
                   (crdt--end-of-block-p left-pos beg-obj) end-obj end-limit)
     (rplacd (get-text-property left-pos 'crdt-id beg-obj) nil) ;; clear end-of-block flag
     t))

;; The protocol
;; Text-based version
;; (it should be easy to migrate to a binary version. Using text for better debugging for now)
;; Every message takes the form (type . body)
;; type can be: insert hello sync
;; - insert
;;   body takes the form (crdt-id position-hint content)
;;   - position-hint is the buffer position where the operation happens at the site
;;     which generates the operation. Then we can play the trick that start search
;;     near this position at other sites to speedup crdt-id search
;;   - content is the string to be inserted
;; - delete
;;   body takes the form (position-hint (crdt-id . length)*)
;; - cursor
;;   body takes the form (site-id point-position-hint point-crdt-id mark-position-hint mark-crdt-id)
;;   *-crdt-id can be either a CRDT ID string, or
;;     - nil, which means clear the cursor/mark
;;     - t, which means end of buffer
;; - hello
;;   This message is sent from client to server, when a client connect to the server.
;;   body is currently nil
;; - sync
;;   This message is sent from server to client to get it sync to the state on the server.
;;   It's always sent after server receives a hello message.
;;   Might be used for error recovery or other optimization in the future.
;;   One optimization I have in mind is let server try to merge all CRDT item into a single
;;   one and try to synchronize this state to clients at best effort.
;;   body takes the form (site-id content . crdt-id-list)
;;   - site-id is the site ID the server assigned to the client
;;   - content is the string in the buffer
;;   - crdt-id-list is generated from CRDT--DUMP-IDS

(defsubst crdt--same-base-p (a b)
  (let* ((a-length (string-bytes a))
         (b-length (string-bytes b)))
    (and (eq a-length b-length)
         (let ((base-length (- a-length 2)))
           (eq t (compare-strings a 0 base-length b 0 base-length))))))
(defun crdt--local-insert (beg end)
  "To be called after a local insert happened in current buffer, from BEG to END.
Returns a list of (insert type) messages to be sent."
  (let (resulting-commands)
    (crdt--with-insertion-information
     (beg end)
     (unless (crdt--split-maybe)
       (when (and not-begin
                  (eq (crdt--id-site starting-id) crdt--local-id)
                  (crdt--end-of-block-p left-pos))
         ;; merge crdt id block
         (let* ((max-offset crdt--max-value)
                (merge-end (min end (+ (- max-offset left-offset 1) beg))))
           (unless (= merge-end beg)
             (put-text-property beg merge-end 'crdt-id starting-id-pair)
             (let ((virtual-id (substring starting-id)))
               (crdt--set-id-offset virtual-id (1+ left-offset))
               (push `(insert ,virtual-id ,beg
                              ,(buffer-substring-no-properties beg merge-end))
                     resulting-commands))
             (setq beg merge-end)))))
     (while (< beg end)
       (let ((block-end (min end (+ crdt--max-value beg))))
         (let ((new-id (crdt--generate-id starting-id left-offset ending-id right-offset crdt--local-id)))
           (put-text-property beg block-end 'crdt-id (cons new-id t))
           (push `(insert ,new-id ,beg
                          ,(buffer-substring-no-properties beg block-end))
                 resulting-commands)
           (setq beg block-end)
           (setq left-offset (1- crdt--max-value)) ; this is always true when we need to continue
           (setq starting-id new-id)))))
    (crdt--verify-buffer)
    (nreverse resulting-commands)))

(defun crdt--find-id (id pos)
  (let* ((left-pos (previous-single-property-change (if (< pos (point-max)) (1+ pos) pos)
                                                    'crdt-id nil (point-min)))
         (left-id (crdt--get-starting-id left-pos))
         (right-pos (next-single-property-change pos 'crdt-id nil (point-max)))
         (right-id (crdt--get-starting-id right-pos)))
    (print (list left-pos left-id right-pos right-id))
    (cl-block nil
      (while t
        (cond ((<= right-pos (point-min))
               (cl-return (point-min)))
              ((>= left-pos (point-max))
               (cl-return (point-max)))
              ((and right-id (not (string< id right-id)))
               (setq left-pos right-pos)
               (setq left-id right-id)
               (setq right-pos (next-single-property-change right-pos 'crdt-id nil (point-max)))
               (setq right-id (crdt--get-starting-id right-pos)))
              ((string< id left-id)
               (setq right-pos left-pos)
               (setq right-id left-id)
               (setq left-pos (previous-single-property-change left-pos 'crdt-id nil (point-min)))
               (setq left-id (crdt--get-starting-id left-pos)))
              (t
               ;; will unibyte to multibyte conversion cause any problem?
               (cl-return
                (if (eq t (compare-strings left-id 0 (- (string-bytes left-id) 2)
                                           id 0 (- (string-bytes left-id) 2)))
                    (min right-pos (+ left-pos 1
                                      (- (crdt--get-two-bytes id (- (string-bytes left-id) 2))
                                         (crdt--id-offset left-id))))
                  right-pos))))))))
(defun crdt--remote-insert (message)
  (let ((crdt--inhibit-update t))
    (cl-destructuring-bind (id position-hint content) message
      (let ((beg (crdt--find-id id position-hint)) end)
        (when beg
          (goto-char beg)
          (insert content)
          (setq end (point))
          (crdt--with-insertion-information
           (beg end)
           (let ((base-length (- (string-bytes starting-id) 2)))
             (if (and (eq (string-bytes id) (string-bytes starting-id))
                      (eq t (compare-strings starting-id 0 base-length
                                             id 0 base-length))
                      (eq (1+ left-offset) (crdt--id-offset id)))
                 (put-text-property beg end 'crdt-id starting-id-pair)
               (put-text-property beg end 'crdt-id (cons id t))))
           (crdt--split-maybe))))))
  (crdt--verify-buffer))

(defun crdt--local-delete (beg end)
  (let ((outer-end end))
    (crdt--with-insertion-information
     (beg 0 nil crdt--changed-string nil (length crdt--changed-string))
     (if (crdt--split-maybe)
         (let* ((not-end (< outer-end (point-max)))
                (ending-id (when not-end (crdt--get-starting-id outer-end))))
           (when (and not-end (eq starting-id (crdt--get-starting-id outer-end)))
             (crdt--set-id outer-end (crdt--id-replace-offset starting-id (+ 1 left-offset (length crdt--changed-string))))
             t))
       (crdt--with-insertion-information
        ((length crdt--changed-string) outer-end crdt--changed-string nil 0 nil)
        (crdt--split-maybe)))))
  (crdt--verify-buffer)
  `(delete ,beg ,@ (crdt--dump-ids 0 (length crdt--changed-string) crdt--changed-string t)))
(defun crdt--remote-delete (message)
  (cl-destructuring-bind (position-hint . id-pairs) message
    (dolist (id-pair id-pairs)
      (cl-destructuring-bind (length . id) id-pair
        (while (> length 0)
          (goto-char (1- (crdt--find-id id position-hint)))
          (let* ((end-of-block (next-single-property-change (point) 'crdt-id nil (point-max)))
                 (block-length (- end-of-block (point))))
            (cl-case (cl-signum (- length block-length))
              ((1) (delete-char block-length)
               (cl-decf length block-length)
               (crdt--set-id-offset id (+ (crdt--id-offset id) block-length)))
              ((0) (delete-char length)
               (setq length 0))
              ((-1)
               (let* ((starting-id (crdt--get-starting-id (point)))
                      (left-offset (crdt--get-id-offset starting-id (point))))
                 (delete-char length)
                 (crdt--set-id (point) (crdt--id-replace-offset starting-id (+ left-offset length))))
               (setq length 0)))))
        (crdt--verify-buffer)))))

(defvar crdt--changed-string nil)
(defun crdt--before-change (beg end)
  (unless crdt--inhibit-update
    (setq crdt--changed-string (buffer-substring beg end))))

(defun crdt--after-change (beg end length)
  (unless crdt--inhibit-update
    (let ((crdt--inhibit-update t))
      ;; we're only interested in text change
      ;; ignore property only changes
      (save-excursion
        (goto-char beg)
        (unless (and (= length (- end beg)) (looking-at (regexp-quote crdt--changed-string)))
          (widen)
          (unless (= length 0)
            (crdt--broadcast-maybe
             (format "%S" (let ((m (crdt--local-delete beg end)))
                            (print m) m))))
          (unless (= beg end)
            (dolist (message (crdt--local-insert beg end))
              (crdt--broadcast-maybe
               (format "%S" (progn (print message) message))))))))))

(defun crdt--dump-ids (beg end object &optional omit-end-of-block-p)
  "Serialize all CRDT ids in OBJECT from BEG to END into a list of
CONSes of the form (LENGTH CRDT-ID . END-OF-BLOCK-P),
or (LENGTH . CRDT-ID) if OMIT-END-OF-BLOCK-P is non-NIL.
in the order that they appears in the document"
  (let (ids (pos end))
    (while (> pos beg)
      (let ((prev-pos (previous-single-property-change pos 'crdt-id object beg)))
        (push (cons (- pos prev-pos)
                    (if omit-end-of-block-p
                        (crdt--get-starting-id prev-pos object)
                      (crdt--get-crdt-id-pair prev-pos object)))
              ids)
        (setq pos prev-pos)))
    ids))
(defun crdt--load-ids (ids)
  "Load the CRDT ids in IDS (generated by CRDT--DUMP-IDS)
into current buffer."
  (let ((pos (point-min)))
    (dolist (id-pair ids)
      (let ((next-pos (+ pos (car id-pair))))
        (put-text-property pos next-pos 'crdt-id (cdr id-pair))
        (setq pos next-pos)))))
(defun crdt--verify-buffer ()
  "Debug helper function to verify that CRDT IDs in a document follows
ascending order."
  (let* ((pos (point-min))
         (id (crdt--get-starting-id pos)))
    (cl-block
        (while t
          (let* ((next-pos (next-single-property-change pos 'crdt-id))
                 (next-id (if (< next-pos (point-max))
                              (crdt--get-starting-id next-pos)
                            (cl-return)))
                 (prev-id (substring id)))
            (crdt--set-id-offset id (+ (- next-pos pos) (crdt--id-offset id)))
            (unless (string< prev-id next-id)
              (error "Not monotonic!"))
            (setq pos next-pos)
            (setq id next-id))))))

(defvar crdt--network-process)
(defvar crdt--network-clients)
(defvar crdt--next-client-id)
(cl-defun crdt--broadcast-maybe (message-string &optional (without t))
  "Broadcast or send MESSAGE-STRING depends on whether CRDT--NETWORK-PROCESS
is a server process.
If CRDT--NETWORK-PROCESS is a server process, broadcast MESSAGE-STRING
to clients except the one of which CLIENT-ID property is EQ to WITHOUT.
If CRDT--NETWORK-PROCESS is a server process, send MESSAGE-STRING
to server unless WITHOUT is NIL."
  (if (process-contact crdt--network-process :server)
      (dolist (client crdt--network-clients)
        (when (and (eq (process-status client) 'open)
                   (not (eq (process-get client 'client-id) without)))
          (process-send-string client message-string)))
    (when without
      (process-send-string crdt--network-process message-string))))
(defun crdt--network-filter (process string)
  (unless (process-buffer process)
    (set-process-buffer process (generate-new-buffer "*crdt-server*"))
    (set-marker (process-mark process) 1))
  (when (buffer-live-p (process-buffer process))
    (with-current-buffer (process-buffer process)
      (when enable-multibyte-characters
        (set-buffer-multibyte nil))
      (save-excursion
        (goto-char (process-mark process))
        (insert string)
        (set-marker (process-mark process) (point))
        (goto-char (point-min))
        (let (message)
          (while (setq message (ignore-errors (read (current-buffer))))
            (print (list 'received message))
            (with-current-buffer (process-get process 'crdt-buffer)
              (save-excursion
                (widen)
                (cl-destructuring-bind (type . body) message
                  (let ((crdt--inhibit-update t))
                    (cl-case type
                      ((insert)
                       (unless (eq (crdt--id-site (car body)) crdt--local-id)
                         (crdt--remote-insert body)
                         (crdt--broadcast-maybe (format "%S" message) (process-get process 'client-id))))
                      ((delete) (crdt--remote-delete body)
                       (crdt--broadcast-maybe (format "%S" message) (process-get process 'client-id)))
                      ((hello) (cl-pushnew process crdt--network-clients)
                       (process-send-string process (format "%S" `(sync
                                                                   ,crdt--next-client-id
                                                                   ,(buffer-substring-no-properties (point-min) (point-max))
                                                                   ,@ (crdt--dump-ids (point-min) (point-max) nil))))
                       (process-put process 'client-id crdt--next-client-id)
                       (cl-incf crdt--next-client-id))
                      ((sync) (erase-buffer)
                       (cl-destructuring-bind (id content . ids) body
                         (insert content)
                         (setq crdt--local-id id)
                         (crdt--load-ids ids))))))))
            (delete-region (point-min) (point))
            (goto-char (point-min))))))))
(defun crdt-serve-buffer (port)
  ""
  (interactive "nPort: ")
  (crdt-mode)
  (setq crdt--local-id 0)
  (setq crdt--network-clients nil)
  (setq crdt--local-clock 0)
  (setq crdt--next-client-id 1)
  (save-excursion
    (widen)
    (let ((crdt--inhibit-update t))
      (crdt--local-insert (point-min) (point-max))))
  (setq crdt--network-process
        (make-network-process
         :name "CRDT Server"
         :server t
         :family 'ipv4
         :host "0.0.0.0"
         :service port
         :filter 'crdt--network-filter
         :plist `(crdt-buffer ,(current-buffer)))))
(defun crdt-stop-serve-buffer ()
  (interactive)
  (delete-process crdt--network-process)
  (dolist (client crdt--network-clients)
    (when (process-live-p client)
      (delete-process client))
    (when (process-buffer client)
      (kill-buffer (process-buffer client))))
  (setq crdt--network-process nil)
  (setq crdt--network-clients nil)
  (crdt-mode 0))
(defun crdt-connect (address port)
  ""
  (interactive "MAddress: \nnPort: ")
  (switch-to-buffer (generate-new-buffer "CRDT Client"))
  (crdt-mode)
  (setq crdt--network-process
        (make-network-process
         :name "CRDT Client"
         :buffer (generate-new-buffer "*crdt-client*")
         :host address
         :family 'ipv4
         :service port
         :filter 'crdt--network-filter
         :plist `(crdt-buffer ,(current-buffer))))
  (process-send-string crdt--network-process
                       (format "%S" '(hello))))
(defun crdt-test-client ()
  (interactive)
  (crdt-connect "127.0.0.1" 1333))
(defun crdt-test-server ()
  (interactive)
  (crdt-serve-buffer 1333))

(define-minor-mode crdt-mode
  "CRDT mode" nil " CRDT" nil
  (if crdt-mode
      (progn
        (add-hook 'after-change-functions #'crdt--after-change nil t)
        (add-hook 'before-change-functions #'crdt--before-change nil t))
    (remove-hook 'after-change-functions #'crdt--after-change t)
    (remove-hook 'before-change-functions #'crdt--before-change t)))

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 1039 bytes --]

Note that there’s some calls to CRDT-VERIFY-BUFFER
which is just for debug purpose (it scan the whole buffer and 
verify integrity). For a performance evaluation replace
it with an empty function or just comment out those calls.

> On Oct 4, 2020, at 7:59 PM, Qiantan Hong <qhong@mit.edu> wrote:
> 
> I’ve finished the initial work for a CRDT based
> collaborative editing Elisp package. The .el is attached.
> 
> The focus is on implementing the core algorithm
> and evaluating performance, so no cursor indicator,
> username display etc was implemented. However,
> they should be easy to add and not relevant to performance
> concern.
> 
> I’ve tested it with my friends and the performance seems 
> reasonable. Using LogootSplit algorithm, the number of
> text property changes are relatively small.
> 
> M-x crdt-test-server starts a local server at port 1333,
> and M-x crdt-test-client connects to 127.0.0.1:1333.
> Or use crdt-serve-buffer and crdt-connect for arbitrary
> address/port.
> <crdt.el>


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Karl Fogel]

[-- Attachment #1.1: Type: text/plain, Size: 1470 bytes --]

>Note that there’s some calls to CRDT-VERIFY-BUFFER
>which is just for debug purpose (it scan the whole buffer and 
>verify integrity). For a performance evaluation replace
>it with an empty function or just comment out those calls.

Note to those who looking for that string in crdt.el:  The symbol is actually `crdt--verify-buffer'.

Qiantan, I'm very excited that you've started this work!  I'll look at the code try it out right away.

May I suggest putting a copyright notice at the top of the file, so it's clear that this is free software?  Please see the attached patch for an example of how to do it.

Best regards,
-Karl

>> On Oct 4, 2020, at 7:59 PM, Qiantan Hong <qhong@mit.edu> wrote:
>> 
>> I’ve finished the initial work for a CRDT based
>> collaborative editing Elisp package. The .el is attached.
>> 
>> The focus is on implementing the core algorithm
>> and evaluating performance, so no cursor indicator,
>> username display etc was implemented. However,
>> they should be easy to add and not relevant to performance
>> concern.
>> 
>> I’ve tested it with my friends and the performance seems 
>> reasonable. Using LogootSplit algorithm, the number of
>> text property changes are relatively small.
>> 
>> M-x crdt-test-server starts a local server at port 1333,
>> and M-x crdt-test-client connects to 127.0.0.1:1333.
>> Or use crdt-serve-buffer and crdt-connect for arbitrary
>> address/port.
>> <crdt.el>


[-- Attachment #1.2: crdt-provenance-patch.txt --]
[-- Type: text/plain, Size: 1079 bytes --]

--- crdt.el	2020-10-04 19:59:06.345935516 -0500
+++ crdt.el	2020-10-04 20:06:29.778398189 -0500
@@ -1,3 +1,24 @@
+;;; crdt.el: collaborative editing using Conflict-free Replicated Data Types
+;; 
+;; Copyright (C) 2020 Qiantan Hong
+;; 
+;; Author: Qiantan Hong <qhong@mit.edu>
+;; Maintainer: Qiantan Hong <qhong@mit.edu>
+;; Keywords: collaboration crdt
+;; 
+;; CRDT is is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+;; 
+;; CRDT is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+;; 
+;; You should have received a copy of the GNU General Public License
+;; along with CRDT.  If not, see <https://www.gnu.org/licenses/>.
+
 (require 'cl-lib)
 
 ;; Loogoot split algorithm

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Last week a friend and I had to write together a text document for a
  > university class. We used google meet to talk and google docs to work on
  > the text document at the same time. That was very efficient and we
  > finished our work in just a couple of hours. This use case is very
  > common in my university.   

Submission to Google may indeed be very convenient -- but I wtill
wouldn't do it.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

crdt.el - collaborative editing for Emacs

[Jean Louis]

Hello Qiantan,

Genious!! That is great, thank you. I have tested it.

It was very easy to start editing, to expose the buffer for
collaboration. 

Could you polish the code, then include the license and propose to
Emacs developers to include it in main stream?

I guess that those that need collaboration will need to start main
Emacs instance on the Internet server. It should be possible to start
remote instance by using ssh, without controlling in real time the
remote instance.

It looks very similar to Gobby editor.

There shall be display of users connected. I did not connect more than
one user.

Jean

* Qiantan Hong <qhong@mit.edu> [2020-10-05 03:00]:
> I’ve finished the initial work for a CRDT based
> collaborative editing Elisp package. The .el is attached.
> 
> The focus is on implementing the core algorithm
> and evaluating performance, so no cursor indicator,
> username display etc was implemented. However,
> they should be easy to add and not relevant to performance
> concern.
> 
> I’ve tested it with my friends and the performance seems 
> reasonable. Using LogootSplit algorithm, the number of
> text property changes are relatively small.
> 
> M-x crdt-test-server starts a local server at port 1333,
> and M-x crdt-test-client connects to 127.0.0.1:1333.
> Or use crdt-serve-buffer and crdt-connect for arbitrary
> address/port.

Re: Question collaborative editing - Wikipedia reference

[Jean Louis]

* Qiantan Hong <qhong@mit.edu> [2020-10-05 03:00]:
> I’ve finished the initial work for a CRDT based
> collaborative editing Elisp package. The .el is attached.

Proposal for crdt.el improvement, for collaborative real-time editing
in Emacs:

- polish the code, propose it to main stream Emacs inclusion, add the
  license 

- let each user customize their username in collaborative
  client-server connection, just as Gobby editor is doing it. Without
  such feature none of the users will know who is the other user.

- make sure that it can be started remotely on public Internet server
  through SSH client, where each of users would then connected. (Those
  who start it in local network or through VPN would know how to do
  it.)

- make sure that it can also start through Tor

- enable secure connection or any kind of other encryption of the
  stream, as otherwise data is leaking, Gobby editor is using locally
  generated certificates

- enable server to have password, so that not every user without
  password can connect, just as Gobby editor is doing it.

Jean

Re: Question collaborative editing - Wikipedia reference

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 2665 bytes --]

> - let each user customize their username in collaborative
>  client-server connection, just as Gobby editor is doing it. Without
>  such feature none of the users will know who is the other user.
Thanks for your suggestion!
I’m thinking of grabbing some code from https://github.com/zk-phi/togetherly/
It uses different color to distinguish users. I can probably then add some
way to toggle displaying full user name with some hotkey to toggle.
Also need to add a list of connected users and color.

I know that emacs’s make-network-process handles TLS if acting as client,
but seems that the server doesn’t. Any idea? 
Or we could rely on some TLS proxy.

> - make sure that it can be started remotely on public Internet server
>  through SSH client, where each of users would then connected. (Those
>  who start it in local network or through VPN would know how to do
>  it.)
> 
> - make sure that it can also start through Tor
Haven't test it. The current stuff sends S-exps over TCP, anybody
knows any potential problem interacting with VPN/Tor?

> - enable server to have password, so that not every user without
>  password can connect, just as Gobby editor is doing it.
Does it have one single temporary password of a server or
instead a user-password model? If it’s the former then is it reasonable
to just send plain password (over TLS connection)? Then that will
be quick to implement.

> On Oct 5, 2020, at 12:02 AM, Jean Louis <bugs@gnu.support> wrote:
> 
> * Qiantan Hong <qhong@mit.edu> [2020-10-05 03:00]:
>> I’ve finished the initial work for a CRDT based
>> collaborative editing Elisp package. The .el is attached.
> 
> Proposal for crdt.el improvement, for collaborative real-time editing
> in Emacs:
> 
> - polish the code, propose it to main stream Emacs inclusion, add the
>  license 
> 
> - let each user customize their username in collaborative
>  client-server connection, just as Gobby editor is doing it. Without
>  such feature none of the users will know who is the other user.
> 
> - make sure that it can be started remotely on public Internet server
>  through SSH client, where each of users would then connected. (Those
>  who start it in local network or through VPN would know how to do
>  it.)
> 
> - make sure that it can also start through Tor
> 
> - enable secure connection or any kind of other encryption of the
>  stream, as otherwise data is leaking, Gobby editor is using locally
>  generated certificates
> 
> - enable server to have password, so that not every user without
>  password can connect, just as Gobby editor is doing it.
> 
> Jean


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Karl Fogel]

[-- Attachment #1: Type: text/plain, Size: 970 bytes --]

Qiantan, I just tried out your code and I'm thrilled -- it worked!  By "worked", I mean:

1) In my current Emacs instance, I loaded crdt.el, switched to a new
   buffer "foo", and ran `M-x crdt-test-server'.

2) Then I started a separate Emacs instance, loaded crdt.el, and ran
   `M-x crdt-test-client'.

3) From both Emacsen, any edit I made was immediately reflected in the
   other Emacs.  That's very promising!

I couldn't easily test simultaneous editing, of course, since there's only one of me, but tomorrow I'll test that with a friend.

Are you doing this work in a public version-controlled repository?  If not, would you like to?  I'd be happy to provide one.  (There are many places that offer public Git hosting, of course; alas, not all of them run on free software, but the one I provide does.)

While crdt.el is obviously still in an early stage, it's very promising!  I'm happy to help test, and am sure others here would be too.

Best regards,
-Karl

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Jean Louis]

* Qiantan Hong <qhong@mit.edu> [2020-10-05 07:11]:
> > - let each user customize their username in collaborative
> >  client-server connection, just as Gobby editor is doing it. Without
> >  such feature none of the users will know who is the other user.
> Thanks for your suggestion!
> I’m thinking of grabbing some code from https://github.com/zk-phi/togetherly/
> It uses different color to distinguish users. I can probably then add some
> way to toggle displaying full user name with some hotkey to toggle.
> Also need to add a list of connected users and color.

I do not think that user names should be disturbing much, but it
should be displayed, maybe in the minibuffer, that user "Qiantan" is
editing currently the text.

Other idea is that at least temporarily, the text edited by certain
user could have specific color of the user. Colors should be assign by
the server. Color would disappear as soon as user stops editing.

> > - make sure that it can also start through Tor
> Haven't test it. The current stuff sends S-exps over TCP, anybody
> knows any potential problem interacting with VPN/Tor?

Maybe that is too much at once.

> > - enable server to have password, so that not every user without
> >  password can connect, just as Gobby editor is doing it.
> Does it have one single temporary password of a server or
> instead a user-password model? If it’s the former then is it reasonable
> to just send plain password (over TLS connection)? Then that will
> be quick to implement.

I think that there should be single password to enter single editing
session, when editing session starts, password should be requested or
customized as default.

Re: Question collaborative editing - Wikipedia reference

[Jean Louis]

 Karl Fogel <kfogel@red-bean.com> [2020-10-05 07:41]:
> Are you doing this work in a public version-controlled repository?
> If not, would you like to?  I'd be happy to provide one.  (There are
> many places that offer public Git hosting, of course; alas, not all
> of them run on free software, but the one I provide does.)

Why not straight to be included in Emacs development git?

I have tried the crdt.el while editing together over network,
collaboratively with two people in real time, and when I yanked text,
I think it did not appear on other side, we were editing together. So
I think yanking does not work.

Re: Question collaborative editing - Wikipedia reference

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> Date: Sun, 4 Oct 2020 23:59:33 +0000
> Cc: Ergus <spacibba@aol.com>, Fermin <fmfs@posteo.net>,
>  Caio Henrique <caiohcs0@gmail.com>, Noam Postavsky <npostavs@gmail.com>,
>  Emacs developers <emacs-devel@gnu.org>, Karl Fogel <kfogel@red-bean.com>,
>  Stefan Monnier <monnier@iro.umontreal.ca>, Eli Zaretskii <eliz@gnu.org>
> 
> I’ve tested it with my friends and the performance seems 
> reasonable. Using LogootSplit algorithm, the number of
> text property changes are relatively small.

Can you show some numbers?  How small is "relatively small", and what
was the size of the buffer you measured that in and the number of
different sites that participated in the collaboration?  Also, did you
benchmark redisplay, say, scrolling the entire buffer, with and
without those text properties, and if so, what are the benchmark
results?

Thanks for working on this.

Re: Question collaborative editing - Wikipedia reference

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> CC: Caio Henrique <caiohcs0@gmail.com>, Ergus <spacibba@aol.com>,
>         "Eli
>  Zaretskii" <eliz@gnu.org>, Fermin <fmfs@posteo.net>,
>         Noam Postavsky
> 	<npostavs@gmail.com>,
>         Emacs developers <emacs-devel@gnu.org>,
>         Karl Fogel
> 	<kfogel@red-bean.com>,
>         Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 5 Oct 2020 04:11:11 +0000
> 
> I’m thinking of grabbing some code from https://github.com/zk-phi/togetherly/

Please be aware that when you use someone else's code, you'd need that
person's permission to use it in Emacs, and we will probably need that
person to assign copyright to the FSF (assuming the borrowed code is
more than ~15 non-trivial lines).

Re: Question collaborative editing - Wikipedia reference

[Stefan Monnier]

> - polish the code, propose it to main stream Emacs inclusion, add the license

I don't see why this should be in Emacs itself rather than in a GNU
ELPA package.


        Stefan

Re: Question collaborative editing - Wikipedia reference

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 895 bytes --]

> Why not straight to be included in Emacs development git?
> 
> I have tried the crdt.el while editing together over network,
> collaboratively with two people in real time, and when I yanked text,
> I think it did not appear on other side, we were editing together. So
> I think yanking does not work.
Thanks for you report!
What I’m award of is currently undo-related stuff won’t work — just haven’t
coded for them. Yanking should in general work, at least when I tested
on my side it does, but it is very possible that there’s some bug that
makes the emacsen fail to sync correctly — implementing distributed
system is hard! I’ll try to reproduce the bug on my side.

A md5 check and resynchronization mechanism can also be added
as a last resort just in case bug happens — and it can make the
system very robust. But let me try to find and fix that bug first.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 05 Oct 2020 02:15:46 -0400
> Cc: Qiantan Hong <qhong@mit.edu>, Fermin <fmfs@posteo.net>,
>  Caio Henrique <caiohcs0@gmail.com>, Ergus <spacibba@aol.com>,
>  Noam Postavsky <npostavs@gmail.com>, Karl Fogel <kfogel@red-bean.com>,
>  Eli Zaretskii <eliz@gnu.org>, Emacs developers <emacs-devel@gnu.org>
> 
> > - polish the code, propose it to main stream Emacs inclusion, add the license
> 
> I don't see why this should be in Emacs itself rather than in a GNU
> ELPA package.

I don't mind having it in either place, FWIW, but AFAICT the package
still needs quite a lot of work before it can be published in either
of them.

Re: Question collaborative editing - Wikipedia reference

[Ergus via Emacs development discussions.]

On October 5, 2020 8:51:24 AM GMT+02:00, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Date: Mon, 05 Oct 2020 02:15:46 -0400
>> Cc: Qiantan Hong <qhong@mit.edu>, Fermin <fmfs@posteo.net>,
>>  Caio Henrique <caiohcs0@gmail.com>, Ergus <spacibba@aol.com>,
>>  Noam Postavsky <npostavs@gmail.com>, Karl Fogel
><kfogel@red-bean.com>,
>>  Eli Zaretskii <eliz@gnu.org>, Emacs developers <emacs-devel@gnu.org>
>> 
>> > - polish the code, propose it to main stream Emacs inclusion, add
>the license
>> 
>> I don't see why this should be in Emacs itself rather than in a GNU
>> ELPA package.
>
>I don't mind having it in either place, FWIW, but AFAICT the package
>still needs quite a lot of work before it can be published in either
>of them.

Of course it does need work. He started it from scratch almost a week ago; it is surprising he brought something working so quickly. Still benchmarks and security improves are needed; maybe use JSON to serialize info... But he needs some more time 

I think that elpa is a better place to have this ATM. But maybe he can host it somewhere in savannah to start with? So the others could start collaborating with him in parallel? I don't know the procedure for that.


Ergus

crdt.el collaborative real time editing for Emacs

[Jean Louis]

Hello Qiantan,

I have the Ethernet switch and few computers in house, I can test
features.

* Qiantan Hong <qhong@mit.edu> [2020-10-05 09:45]:
> Hi Jean,
> 
> Thanks again for the report! Currently the code is at very
> alpha stage so sorry for any bugs in it.

That is natural.

> Could you provide a bit more detail on the yanking issue?

Yes, I have been testing that, first it appeared to work, then when
yanked from server, it did not appear on the connected computer.

Then I have saved file on server. Then I have yanked again into file,
it did not appear on connected computer.

> Did it just failed for one particular yank operation or all yanking
> doesn’t work?

Now I cannot say why it happens, I say it happens.

If undo is done on connected computer, the undo is not shown on the
server.

> I’m more worried about the former case — which means some
> bug lies in the algorithm implementation and it’s really hard to
> reproduce — haven’t reproduce it yet.

> It would be also helpful in this case that you can provide some
> description of operations happened before this bug is triggered.

> But if it’s the latter then it might just be some other package
> interference with the change hook or the change hook is not
> installed correctly, either case, much easier to fix.

Sure, I will do that little later, when I have computers one to each
other, right now they are separate, I have to walk from room to room
and do action by action.

What means crdt.el ?

Many times of editing are collaborative editing, to say that it is
only collaborative would not be proper, it is better to say "real time
collaboration" and even better "simultaneous editing":
https://en.wikipedia.org/wiki/Simultaneous_editing

The mode should be named in easier fashion, something that is easy for
users to enable or turn off.

Later I will give you more detailed transcript what I did, the crdt.el
leaves some messages too, I can collect those as well. 

Jean

Re: crdt.el collaborative real time editing for Emacs

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 3059 bytes --]

Thanks! Actually, it will be incredibly useful if you can get a dump
of CRDT ID state. To do this, M-: (set-buffer-multibyte nil) in 
*Message* (to make it unibyte, CRDT ID are arbitrary binary strings
in general and multibyte buffer does some unicode conversion stuff),
and then M-: (crdt--dump-ids) in the CRDT server/client buffers. This
basically dump out the complete state of CRDT IDs and
should be easy to spot inconsistency between dumps on different
emacsen. Note that it might be very long and get truncated by *Message*.
M-: (setq some-var (crdt--dump-ids)) in the CRDT buffer and then
M-: (insert some-var) in a unibyte buffer is a workaround. Or
any other method that you prefer.

> If undo is done on connected computer, the undo is not shown on the
> server.
Yep haven’t work on undo-redo at all. Any behavior can happen.

> On Oct 5, 2020, at 4:07 AM, Jean Louis <bugs@gnu.support> wrote:
> 
> Hello Qiantan,
> 
> I have the Ethernet switch and few computers in house, I can test
> features.
> 
> * Qiantan Hong <qhong@mit.edu> [2020-10-05 09:45]:
>> Hi Jean,
>> 
>> Thanks again for the report! Currently the code is at very
>> alpha stage so sorry for any bugs in it.
> 
> That is natural.
> 
>> Could you provide a bit more detail on the yanking issue?
> 
> Yes, I have been testing that, first it appeared to work, then when
> yanked from server, it did not appear on the connected computer.
> 
> Then I have saved file on server. Then I have yanked again into file,
> it did not appear on connected computer.
> 
>> Did it just failed for one particular yank operation or all yanking
>> doesn’t work?
> 
> Now I cannot say why it happens, I say it happens.
> 
> If undo is done on connected computer, the undo is not shown on the
> server.
> 
>> I’m more worried about the former case — which means some
>> bug lies in the algorithm implementation and it’s really hard to
>> reproduce — haven’t reproduce it yet.
> 
>> It would be also helpful in this case that you can provide some
>> description of operations happened before this bug is triggered.
> 
>> But if it’s the latter then it might just be some other package
>> interference with the change hook or the change hook is not
>> installed correctly, either case, much easier to fix.
> 
> Sure, I will do that little later, when I have computers one to each
> other, right now they are separate, I have to walk from room to room
> and do action by action.
> 
> What means crdt.el ?
> 
> Many times of editing are collaborative editing, to say that it is
> only collaborative would not be proper, it is better to say "real time
> collaboration" and even better "simultaneous editing":
> https://en.wikipedia.org/wiki/Simultaneous_editing
> 
> The mode should be named in easier fashion, something that is easy for
> users to enable or turn off.
> 
> Later I will give you more detailed transcript what I did, the crdt.el
> leaves some messages too, I can collect those as well. 
> 
> Jean


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Jean Louis]

* Stefan Monnier <monnier@iro.umontreal.ca> [2020-10-05 09:16]:
> > - polish the code, propose it to main stream Emacs inclusion, add the license
> 
> I don't see why this should be in Emacs itself rather than in a GNU
> ELPA package.

For users to easier access real time simultaneous collaborative
editing features.

crdt.el collaborative real time editing for Emacs

[Jean Louis]

* Qiantan Hong <qhong@mit.edu> [2020-10-05 09:19]:
> > Why not straight to be included in Emacs development git?
> > 
> > I have tried the crdt.el while editing together over network,
> > collaboratively with two people in real time, and when I yanked text,
> > I think it did not appear on other side, we were editing together. So
> > I think yanking does not work.
> Thanks for you report!

> What I’m award of is currently undo-related stuff won’t work — just
> haven’t coded for them.

I think that you could only take what is being replaced or inserted
into buffer, you need not carry of undo on Emacs client side, just
what is in buffer.

As each client, just as in Gobby real time collaborative editor, will
be able to undo for themselves individually. I hope you get me.

Re: crdt.el collaborative real time editing for Emacs

[Eli Zaretskii]

> Date: Mon, 5 Oct 2020 11:07:26 +0300
> From: Jean Louis <bugs@gnu.support>
> Cc: emacs-devel@gnu.org
> 
> What means crdt.el ?

Conflict-free Replicated Data Type.

Re: Question collaborative editing - Wikipedia reference

[Stefan Monnier]

>> > - polish the code, propose it to main stream Emacs inclusion, add the license
>> I don't see why this should be in Emacs itself rather than in a GNU
>> ELPA package.
> For users to easier access real time simultaneous collaborative
> editing features.

I don't see why that should motivate putting it into emacs.git rather
than elpa.git.


        Stefan

Re: crdt.el collaborative real time editing for Emacs

[Jean Louis]

I have noticed, if the file is no longer edited, the process for CRDT,
collaboration in real time remains in memory, it can be seen with
list-processes, and it prevents opening a new file to be shared.

It should be possible to open multiple files to be shared for real
time simultaneous editing, that would invoke multiple servers tied to
multiple buffers.

And when buffer is no longer there, server on that port should stop I
guess.

Re: crdt.el collaborative real time editing for Emacs

[Jean Louis]

* Qiantan Hong <qhong@mit.edu> [2020-10-05 11:22]:
> Thanks! Actually, it will be incredibly useful if you can get a dump
> of CRDT ID state. To do this, M-: (set-buffer-multibyte nil) in 
> *Message* (to make it unibyte, CRDT ID are arbitrary binary strings
> in general and multibyte buffer does some unicode conversion stuff),
> and then M-: (crdt--dump-ids) in the CRDT server/client buffers. This
> basically dump out the complete state of CRDT IDs and
> should be easy to spot inconsistency between dumps on different
> emacsen. Note that it might be very long and get truncated by *Message*.
> M-: (setq some-var (crdt--dump-ids)) in the CRDT buffer and then
> M-: (insert some-var) in a unibyte buffer is a workaround. Or
> any other method that you prefer.

I have tried, I will do that process after you do more polishing.

For now, I can say I have copied tutorial, yanked it, it did not
appear on other side. Smaller yanks worked.

Re: crdt.el collaborative real time editing for Emacs

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 996 bytes --]

Yes, definitely need to work on that. 
Just need some process sentinel and make the crdt—* variables 
buffer local.

Also in terms of yanking problem, I tried with crdt.el itself
and didn’t reproduce the bug. I then proceed to yank the whole emacs
CHANGELOG.3 into it (that’s 100k lines!) and it’s slowly going
through while almost hanging the emacsen — not sure if
this is what you observed. Shall we discuss more in detail
off list?
> On Oct 5, 2020, at 10:03 AM, Jean Louis <bugs@gnu.support> wrote:
> 
> I have noticed, if the file is no longer edited, the process for CRDT,
> collaboration in real time remains in memory, it can be seen with
> list-processes, and it prevents opening a new file to be shared.
> 
> It should be possible to open multiple files to be shared for real
> time simultaneous editing, that would invoke multiple servers tied to
> multiple buffers.
> 
> And when buffer is no longer there, server on that port should stop I
> guess.


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Karl Fogel]

On 05 Oct 2020, Stefan Monnier wrote:
>>> > - polish the code, propose it to main stream Emacs inclusion, add the license
>>> I don't see why this should be in Emacs itself rather than in a GNU
>>> ELPA package.
>> For users to easier access real time simultaneous collaborative
>> editing features.
>
>I don't see why that should motivate putting it into emacs.git rather
>than elpa.git.

Both of these are options for "endorsed publication".  That is, they involve putting the code in a third-party location in a way that implies some form of (perhaps qualified or conditional) endorsement by that third party.

But I wasn't talking about endorsed publication.  I was just proposing a place with a public git repository, issue tracker, etc, in order to make development and testing easier.  Right now, Qiantan is posting successive versions of crdt.el to a mailing list -- which is not optimal :-).  Having a public version control repository makes it easier for others to follow development closely, offer patches, and so on.

What ELPA and the Emacs tree offer is a superset of that.  Maybe some day crdt.el will be at a point where the stuff in the superset is desirable, but we're not at that stage yet, and I hadn't expected anyone to raise them in response to my suggestion.

Best regards,
-Karl

Re: crdt.el collaborative real time editing for Emacs

[Jean Louis]

* Qiantan Hong <qhong@mit.edu> [2020-10-05 20:06]:
> Yes, definitely need to work on that. 
> Just need some process sentinel and make the crdt—* variables 
> buffer local.
> 
> Also in terms of yanking problem, I tried with crdt.el itself
> and didn’t reproduce the bug. I then proceed to yank the whole emacs
> CHANGELOG.3 into it (that’s 100k lines!) and it’s slowly going
> through while almost hanging the emacsen — not sure if
> this is what you observed. Shall we discuss more in detail
> off list?

I did not observe that, it was quick yank, but I did not see slow
synchronization.

Re: Question collaborative editing - Wikipedia reference

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 1578 bytes --]

> But this has 3 main problems.
> 
> 1) On one hand such services require some servers (to work like google
> spreadsheet) and need to be provided somehow... something difficult as I
> don't think gnu or fsf have resources to maintain a service like that
> and provide it.
> 
> 2) On the other hand it will be better if the service is somehow
> distributed in order to give more privacy-security but also to reduce
> the load of the servers... I still can't find any infrastructure we can
> use, cause most of the peer-to-peer libraries are for C++, javascript,
> Node.js and so on (example: webrtc). Just on yesterday I found
> n2n... But I am not a web specialist so it requires a lot of
> experimenting time for me.
> 
> 3) The other workflow (create a local server for others) is the
> "simplest" approach at the moment. But that is a problem for many use
> cases due to dynamic ip addreses, firewalls, opening ports and so on. It
> is fine for a class room or company, but not for working from home.

On this topic, I’m considering supporting sending the traffic over
IRC. Seems that it solves all those problem, what do you guys think?

The process will be that one user create a channel with a random
name, say on freenode.net, then they share the channel name
with other user (maybe via IRC as well!). Others can then join the
channel, and it behaves basically like TCP. To avoid spamming
the same authentication protocol for TCP (to be implemented) can
also work on IRC. The messages from user without authentication
are simply discarded.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - Wikipedia reference

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 2037 bytes --]

Qiantan Hong <qhong@mit.edu> writes:


Another option might be jabber -- it had many design goals that are
relevant here. Jabber used a light-weight XML serialization which I
suspect is nicely isomorphic to s-expressions in lisp. Once you have a
jabber stream established, the back-and-forth communication are xml
serializations of s-expressions.>> But this has 3 main problems.
>> 
>> 1) On one hand such services require some servers (to work like google
>> spreadsheet) and need to be provided somehow... something difficult as I
>> don't think gnu or fsf have resources to maintain a service like that
>> and provide it.
>> 
>> 2) On the other hand it will be better if the service is somehow
>> distributed in order to give more privacy-security but also to reduce
>> the load of the servers... I still can't find any infrastructure we can
>> use, cause most of the peer-to-peer libraries are for C++, javascript,
>> Node.js and so on (example: webrtc). Just on yesterday I found
>> n2n... But I am not a web specialist so it requires a lot of
>> experimenting time for me.
>> 
>> 3) The other workflow (create a local server for others) is the
>> "simplest" approach at the moment. But that is a problem for many use
>> cases due to dynamic ip addreses, firewalls, opening ports and so on. It
>> is fine for a class room or company, but not for working from home.
>
> On this topic, I¡¯m considering supporting sending the traffic over
> IRC. Seems that it solves all those problem, what do you guys think?
>
> The process will be that one user create a channel with a random
> name, say on freenode.net, then they share the channel name
> with other user (maybe via IRC as well!). Others can then join the
> channel, and it behaves basically like TCP. To avoid spamming
> the same authentication protocol for TCP (to be implemented) can
> also work on IRC. The messages from user without authentication
> are simply discarded.
>

-- 

Thanks,

--Raman
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Question collaborative editing - Wikipedia reference

[Jean Louis]

Don't complicate, use Gobby editor as example, no IRC, XMPP and other networking dependencies please.


On October 6, 2020 1:03:57 AM UTC, Qiantan Hong <qhong@mit.edu> wrote:
>> But this has 3 main problems.
>> 
>> 1) On one hand such services require some servers (to work like
>google
>> spreadsheet) and need to be provided somehow... something difficult
>as I
>> don't think gnu or fsf have resources to maintain a service like that
>> and provide it.
>> 
>> 2) On the other hand it will be better if the service is somehow
>> distributed in order to give more privacy-security but also to reduce
>> the load of the servers... I still can't find any infrastructure we
>can
>> use, cause most of the peer-to-peer libraries are for C++,
>javascript,
>> Node.js and so on (example: webrtc). Just on yesterday I found
>> n2n... But I am not a web specialist so it requires a lot of
>> experimenting time for me.
>> 
>> 3) The other workflow (create a local server for others) is the
>> "simplest" approach at the moment. But that is a problem for many use
>> cases due to dynamic ip addreses, firewalls, opening ports and so on.
>It
>> is fine for a class room or company, but not for working from home.
>
>On this topic, I’m considering supporting sending the traffic over
>IRC. Seems that it solves all those problem, what do you guys think?
>
>The process will be that one user create a channel with a random
>name, say on freenode.net, then they share the channel name
>with other user (maybe via IRC as well!). Others can then join the
>channel, and it behaves basically like TCP. To avoid spamming
>the same authentication protocol for TCP (to be implemented) can
>also work on IRC. The messages from user without authentication
>are simply discarded.


Jean

Re: Question collaborative editing - Wikipedia reference

[Jean Louis]

* Qiantan Hong <qhong@mit.edu> [2020-10-06 04:04]:
> > But this has 3 main problems.
> > 
> > 1) On one hand such services require some servers (to work like google
> > spreadsheet) and need to be provided somehow... something difficult as I
> > don't think gnu or fsf have resources to maintain a service like that
> > and provide it.

Comment on crdt.el in the regard to above quote is that the server is
on Emacs, so there need not be and should not be any third party to
provide any service to provide service to Emacs, as Emacs is providing
server through crdt.el

Please look at how Gobby real time simultaneous editor is working,
that is very similar now to crdt.el

If user have Emacs with public IP address, all such user need to do is
to tell the IP address and port and password to the collaborator, they
can already connect.

If user do not have public IP address but user is behind the router,
then such user who connects over network or administrator who is
helping users to connect over the network is assumed to have
sufficient knowledge how to click few times or setup router to port
forward from local network to public network.

If user has VPS or dedicated server, then such can use ssh to port
forward to the public server, or can use permanent VPN to connect to
other collaborators which is anyway best setup. When I say permanent
VPN, I mean "private" network and not VPN through third parties.

To use VPS, today it is about $5 or less, expenses are not high. 

And let us not forget that there are many organizations where such
collaborative simultaneous editing can take place who need not have
any Internet connection, they may have their hotspots for local
network or ethernet wire networks.

> > 2) On the other hand it will be better if the service is somehow
> > distributed in order to give more privacy-security but also to reduce
> > the load of the servers...

We speak here of editing of a file, not transfer of huge binaries with
many users. So there is no expected high load on the server. I cannot
possibly understand why should simultaneous real time editing be
distributed, maybe you think in similar fashion like Tor is doing it,
but that is all definitely not necessary for Emacs to handle. You can
handle your routing anyway through Tor, I am sure it will work, and
you can route traffic through VPN, SSH and maybe other means. 

> > I still can't find any infrastructure we can
> > use, cause most of the peer-to-peer libraries are for C++, javascript,
> > Node.js and so on (example: webrtc). Just on yesterday I found
> > n2n... But I am not a web specialist so it requires a lot of
> > experimenting time for me.

For crdt.el is definitely not needed, just as in Gobby editor is not
needed. Collaborators who connect over Internet will have enough
networking skills to connect to each other by using methods as I have
explained above. Simplest method is ssh, and if any of collaborators
have public IP exposed, such could be preferred server, and everybody
else could connect to such, by doing simple ssh command with port
forwarding, and editing afterwards.

> > 3) The other workflow (create a local server for others) is the
> > "simplest" approach at the moment. But that is a problem for many use
> > cases due to dynamic ip addreses, firewalls, opening ports and so on. It
> > is fine for a class room or company, but not for working from  home.

All solutions for that already exists, see above.

> On this topic, I'm considering supporting sending the traffic over
> IRC. Seems that it solves all those problem, what do you guys think?

I think I feel like "abused" when I hear that.

> The process will be that one user create a channel with a random
> name, say on freenode.net, then they share the channel name
> with other user (maybe via IRC as well!). Others can then join the
> channel, and it behaves basically like TCP. To avoid spamming
> the same authentication protocol for TCP (to be implemented) can
> also work on IRC. The messages from user without authentication
> are simply discarded.

Please no.

:-)


Jean

Re: Question collaborative editing - Wikipedia reference

[Karl Fogel]

[-- Attachment #1: Type: text/plain, Size: 874 bytes --]

On 06 Oct 2020, Qiantan Hong wrote:
>On this topic, I’m considering supporting sending the traffic over
>IRC. Seems that it solves all those problem, what do you guys think?
>
>The process will be that one user create a channel with a random
>name, say on freenode.net, then they share the channel name
>with other user (maybe via IRC as well!). Others can then join the
>channel, and it behaves basically like TCP. To avoid spamming
>the same authentication protocol for TCP (to be implemented) can
>also work on IRC. The messages from user without authentication
>are simply discarded.

That's an interesting idea!

But do you intend to keep the "no server" option available too?  That is, for people whose network firewalls allow it, will it always be possible to just run crdt.el with one of the Emacsen acting as server itself?

Best regards,
-Karl

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Qiantan Hong]

[-- Attachment #1.1: Type: text/plain, Size: 748 bytes --]

> That's an interesting idea!
> 
> But do you intend to keep the "no server" option available too?  That is, for people whose network firewalls allow it, will it always be possible to just run crdt.el with one of the Emacsen acting as server itself?

Sure of course, that’s the easy option.

Also here is an a-bit-more-polished version.
Now there’s
- cursor/marked region indication
- optional challenge-response authentication
- I think it handles undo correctly now. Previously
it was interfering with the text properties.
- cleanup the dead processes etc correctly

Also for the yanking issue Jeans reported, I haven’t 
reproduce it successfully so it is a bug it will probably 
still be there, but you can give a try. 


[-- Attachment #1.2: crdt.el --]
[-- Type: application/octet-stream, Size: 33808 bytes --]

;;; crdt.el --- collaborative editing using Conflict-free Replicated Data Types
;;
;; Copyright (C) 2020 Qiantan Hong
;;
;; Author: Qiantan Hong <qhong@mit.edu>
;; Maintainer: Qiantan Hong <qhong@mit.edu>
;; Keywords: collaboration crdt
;;
;; CRDT is is free software: you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.
;;
;; CRDT is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.
;;
;; You should have received a copy of the GNU General Public License
;; along with CRDT.  If not, see <https://www.gnu.org/licenses/>.

;;; Commentary:
;; * Algorithm
;;   This packages implements the Logoot split algorithm
;;   André, Luc, et al.  "Supporting adaptable granularity of changes for massive-scale collaborative editing." 9th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing.  IEEE, 2013.
;; * Protocol
;;   Text-based version
;;   (it should be easy to migrate to a binary version.  Using text for better debugging for now)
;;   Every message takes the form (type . body)
;;   type can be: insert hello sync
;;   - insert
;;     body takes the form (crdt-id position-hint content)
;;     - position-hint is the buffer position where the operation happens at the site
;;       which generates the operation.  Then we can play the trick that start search
;;       near this position at other sites to speedup crdt-id search
;;     - content is the string to be inserted
;;   - delete
;;     body takes the form (position-hint (crdt-id . length)*)
;;   - cursor
;;     body takes the form
;;          (site-id point-position-hint point-crdt-id mark-position-hint mark-crdt-id)
;;     *-crdt-id can be either a CRDT ID string, or
;;       - nil, which means clear the point/mark
;;   - hello
;;     This message is sent from client to server, when a client connect to the server.
;;     body takes the form (client-name &optional response)
;;   - challenge
;;     body takes the form (salt)
;;   - sync
;;     This message is sent from server to client to get it sync to the state on the server.
;;     It's always sent after server receives a hello message.
;;     Might be used for error recovery or other optimization in the future.
;;     One optimization I have in mind is let server try to merge all CRDT item into a single
;;     one and try to synchronize this state to clients at best effort.
;;     body takes the form (site-id major-mode content . crdt-id-list)
;;     - site-id is the site ID the server assigned to the client
;;     - major-mode is the major mode used at the server site
;;     - content is the string in the buffer
;;     - crdt-id-list is generated from CRDT--DUMP-IDS


;;; Code:
(require 'cl-lib)

(require 'color)
(defvar crdt-cursor-region-colors
  (let ((n 10))
    (cl-loop for i below n
             for hue by (/ 1.0 n)
             collect (cons
                      (apply #'color-rgb-to-hex
                             (color-hsl-to-rgb hue 0.5 0.5))
                      (apply #'color-rgb-to-hex
                             (color-hsl-to-rgb hue 0.2 0.5))))))

(defun crdt--get-cursor-color (site-id)
  "Get cursor color for SITE-ID."
  (car (nth (mod site-id (length crdt-cursor-region-colors)) crdt-cursor-region-colors)))
(defun crdt--get-region-color (site-id)
  "Get region color for SITE-ID."
  (cdr (nth (mod site-id (length crdt-cursor-region-colors)) crdt-cursor-region-colors)))
(defun crdt--move-cursor (ov pos)
  "Move pseudo cursor overlay OV to POS."
  (let* ((eol (or (eq pos (point-max))
                  (eq (char-after pos) ?\n)))
         (end (if eol pos (1+ pos)))
         (after-str (when eol (propertize " " 'face (overlay-get ov 'face)))))
    (move-overlay ov pos end)
    (overlay-put ov 'after-string after-str)))
(defun crdt--move-region (ov pos mark)
  "Move pseudo marked region overlay OV to mark between POS and MARK."
  (move-overlay ov (min pos mark) (max pos mark)))


;; CRDT IDs are represented by unibyte strings (for efficient comparison)
;; Every two bytes represent a big endian encoded integer
;; For base IDs, last two bytes are always representing site ID
;; Stored strings are BASE-ID:OFFSETs. So the last two bytes represent offset,
;; and second last two bytes represent site ID
(defconst crdt--max-value (lsh 1 16))
;; (defconst crdt--max-value 4)
;; for debug
(defconst crdt--low-byte-mask 255)
(defsubst crdt--get-two-bytes (string index)
  "Get the big-endian encoded integer from STRING starting from INDEX.
INDEX is counted by bytes."
  (logior (lsh (elt string index) 8)
          (elt string (1+ index))))
(defsubst crdt--get-two-bytes-with-offset (string offset index default)
  "Helper function for CRDT--GENERATE-ID.
Get the big-endian encoded integer from STRING starting from INDEX,
but with last two-bytes of STRING (the offset portion) replaced by OFFSET,
and padded infintely by DEFAULT to the right."
  (cond ((= index (- (string-bytes string) 2))
         offset)
        ((< (1+ index) (string-bytes string))
         (logior (lsh (elt string index) 8)
                 (elt string (1+ index))))
        (t default)))

(defsubst crdt--id-offset (id)
  "Get the literal offset integer from ID.
Note that it might deviate from real offset for a character
in the middle of a block."
  (crdt--get-two-bytes id (- (string-bytes id) 2)))
(defsubst crdt--set-id-offset (id offset)
  "Set the OFFSET portion of ID destructively."
  (let ((length (string-bytes id)))
    (aset id (- length 2) (lsh offset -8))
    (aset id (- length 1) (logand offset crdt--low-byte-mask))))
(defsubst crdt--id-replace-offset (id offset)
  "Create and return a new id string by replacing the OFFSET portion from ID."
  (let ((new-id (substring id)))
    (crdt--set-id-offset new-id offset)
    new-id))
(defsubst crdt--id-site (id)
  "Get the site id from ID."
  (crdt--get-two-bytes id (- (string-bytes id) 4)))
(defsubst crdt--generate-id (low-id low-offset high-id high-offset site-id)
  "Generate a new ID between LOW-ID and HIGH-ID.
The generating site is marked as SITE-ID.
Offset parts of LOW-ID and HIGH-ID are overriden by LOW-OFFSET
and HIGH-OFFSET.  (to save two copying from using CRDT--ID-REPLACE-OFFSET)"
  (let* ((l (crdt--get-two-bytes-with-offset low-id low-offset 0 0))
         (h (crdt--get-two-bytes-with-offset high-id high-offset 0 crdt--max-value))
         (bytes (cl-loop for pos from 2 by 2
                         while (< (- h l) 2)
                         append (list (lsh l -8)
                                      (logand l crdt--low-byte-mask))
                         do (setq l (crdt--get-two-bytes-with-offset low-id low-offset pos 0))
                         do (setq h (crdt--get-two-bytes-with-offset high-id high-offset pos crdt--max-value))))
         (m (+ l 1 (random (- h l 1)))))
    (apply #'unibyte-string
           (append bytes (list (lsh m -8)
                               (logand m crdt--low-byte-mask)
                               (lsh site-id -8)
                               (logand site-id crdt--low-byte-mask)
                               0
                               0)))))

;; CRDT-ID text property actually stores a cons of (ID-STRING . END-OF-BLOCK-P)
(defsubst crdt--get-crdt-id-pair (pos &optional obj)
  "Get the (CRDT-ID . END-OF-BLOCK-P) pair at POS in OBJ."
  (get-text-property pos 'crdt-id obj))
(defsubst crdt--get-starting-id (pos &optional obj)
  "Get the CRDT-ID at POS in OBJ."
  (car (crdt--get-crdt-id-pair pos obj)))
(defsubst crdt--end-of-block-p (pos &optional obj)
  "Get the END-OF-BLOCK-P at POS in OBJ."
  (cdr (crdt--get-crdt-id-pair pos obj)))
(defsubst crdt--get-starting-id-maybe (pos &optional obj limit)
  "Get the CRDT-ID at POS in OBJ if POS is no smaller than LIMIT.
Return NIL otherwise."
  (unless (< pos (or limit (point-min)))
    (car (get-text-property pos 'crdt-id obj))))
(defsubst crdt--get-id-offset (starting-id pos &optional obj limit)
  "Get the real offset integer for a character at POS.
Assume the stored literal ID is STARTING-ID."
  (let* ((start-pos (previous-single-property-change (1+ pos) 'crdt-id obj (or limit (point-min)))))
    (+ (- pos start-pos) (crdt--id-offset starting-id))))
(defsubst crdt--get-id (pos &optional obj limit)
  "Get the real CRDT ID at POS."
  (let ((limit (or limit (point-min))))
    (if (< pos limit) ""
      (let* ((starting-id (crdt--get-starting-id pos obj))
             (left-offset (crdt--get-id-offset starting-id pos obj limit)))
        (crdt--id-replace-offset starting-id left-offset)))))

(defsubst crdt--set-id (pos id &optional end-of-block-p obj limit)
  "Set the crdt ID and END-OF-BLOCK-P at POS in OBJ.
Any characters after POS but before LIMIT that used to
have the same (CRDT-ID . END-OF-BLOCK-P) pair are also updated
with ID and END-OF-BLOCK-P."
  (put-text-property pos (next-single-property-change pos 'crdt-id obj (or limit (point-max))) 'crdt-id (cons id end-of-block-p) obj))

(cl-defmacro crdt--with-insertion-information
    ((beg end &optional beg-obj end-obj beg-limit end-limit) &body body)
  "Helper macro to setup some useful variables."
  `(let* ((not-begin (> ,beg ,(or beg-limit '(point-min)))) ; if it's nil, we're at the beginning of buffer
          (left-pos (1- ,beg))
          (starting-id-pair (when not-begin (crdt--get-crdt-id-pair left-pos ,beg-obj)))
          (starting-id (if not-begin (car starting-id-pair) ""))
          (left-offset (if not-begin (crdt--get-id-offset starting-id left-pos ,beg-obj ,beg-limit) 0))
          (not-end (< ,end ,(or end-limit '(point-max))))
          (ending-id (if not-end (crdt--get-starting-id ,end ,end-obj) ""))
          (right-offset (if not-end (crdt--id-offset ending-id) 0))
          (beg ,beg)
          (end ,end)
          (beg-obj ,beg-obj)
          (end-obj ,end-obj)
          (beg-limit ,beg-limit)
          (end-limit ,end-limit))
     ,@body))
(defmacro crdt--split-maybe ()
  '(when (and not-end (eq starting-id (crdt--get-starting-id end end-obj)))
     ;; need to split id block
     (crdt--set-id end (crdt--id-replace-offset starting-id (1+ left-offset))
                   (crdt--end-of-block-p left-pos beg-obj) end-obj end-limit)
     (rplacd (get-text-property left-pos 'crdt-id beg-obj) nil) ;; clear end-of-block flag
     t))

(defsubst crdt--same-base-p (a b)
  (let* ((a-length (string-bytes a))
         (b-length (string-bytes b)))
    (and (eq a-length b-length)
         (let ((base-length (- a-length 2)))
           (eq t (compare-strings a 0 base-length b 0 base-length))))))

(defmacro crdt--defvar-permanent-local (name &optional val docstring)
  `(progn
     (defvar-local ,name ,val ,docstring)
     (put ',name 'permanent-local t)))
(crdt--defvar-permanent-local crdt--local-id nil "Local site-id.")
(crdt--defvar-permanent-local crdt--inhibit-update nil "When set, don't call CRDT--LOCAL-* on change.
This is useful for functions that apply remote change to local buffer,
to avoid recusive calling of CRDT synchronization functions.")
(crdt--defvar-permanent-local crdt--changed-string nil)
(crdt--defvar-permanent-local crdt--last-point nil)
(crdt--defvar-permanent-local crdt--last-mark nil)
(crdt--defvar-permanent-local crdt--overlay-table nil)

(defun crdt--local-insert (beg end)
  "To be called after a local insert happened in current buffer from BEG to END.
Returns a list of (insert type) messages to be sent."
  (let (resulting-commands)
    (crdt--with-insertion-information
     (beg end)
     (unless (crdt--split-maybe)
       (when (and not-begin
                  (eq (crdt--id-site starting-id) crdt--local-id)
                  (crdt--end-of-block-p left-pos))
         ;; merge crdt id block
         (let* ((max-offset crdt--max-value)
                (merge-end (min end (+ (- max-offset left-offset 1) beg))))
           (unless (= merge-end beg)
             (put-text-property beg merge-end 'crdt-id starting-id-pair)
             (let ((virtual-id (substring starting-id)))
               (crdt--set-id-offset virtual-id (1+ left-offset))
               (push `(insert ,virtual-id ,beg
                              ,(buffer-substring-no-properties beg merge-end))
                     resulting-commands))
             (setq beg merge-end)))))
     (while (< beg end)
       (let ((block-end (min end (+ crdt--max-value beg))))
         (let ((new-id (crdt--generate-id starting-id left-offset ending-id right-offset crdt--local-id)))
           (put-text-property beg block-end 'crdt-id (cons new-id t))
           (push `(insert ,new-id ,beg
                          ,(buffer-substring-no-properties beg block-end))
                 resulting-commands)
           (setq beg block-end)
           (setq left-offset (1- crdt--max-value)) ; this is always true when we need to continue
           (setq starting-id new-id)))))
    ;; (crdt--verify-buffer)
    (nreverse resulting-commands)))

(defun crdt--find-id (id pos)
  "Find the first position *after* ID.  Start the search from POS."
  (let* ((left-pos (previous-single-property-change (if (< pos (point-max)) (1+ pos) pos)
                                                    'crdt-id nil (point-min)))
         (left-id (crdt--get-starting-id left-pos))
         (right-pos (next-single-property-change pos 'crdt-id nil (point-max)))
         (right-id (crdt--get-starting-id right-pos)))
    (cl-block nil
      (while t
        (cond ((<= right-pos (point-min))
               (cl-return (point-min)))
              ((>= left-pos (point-max))
               (cl-return (point-max)))
              ((and right-id (not (string< id right-id)))
               (setq left-pos right-pos)
               (setq left-id right-id)
               (setq right-pos (next-single-property-change right-pos 'crdt-id nil (point-max)))
               (setq right-id (crdt--get-starting-id right-pos)))
              ((string< id left-id)
               (setq right-pos left-pos)
               (setq right-id left-id)
               (setq left-pos (previous-single-property-change left-pos 'crdt-id nil (point-min)))
               (setq left-id (crdt--get-starting-id left-pos)))
              (t
               ;; will unibyte to multibyte conversion cause any problem?
               (cl-return
                (if (eq t (compare-strings left-id 0 (- (string-bytes left-id) 2)
                                           id 0 (- (string-bytes left-id) 2)))
                    (min right-pos (+ left-pos 1
                                      (- (crdt--get-two-bytes id (- (string-bytes left-id) 2))
                                         (crdt--id-offset left-id))))
                  right-pos))))))))
(defun crdt--remote-insert (message)
  (let ((crdt--inhibit-update t))
    (cl-destructuring-bind (id position-hint content) message
      (let ((beg (crdt--find-id id position-hint)) end)
        (goto-char beg)
        (insert content)
        (setq end (point))
        (with-silent-modifications
          (crdt--with-insertion-information
           (beg end)
           (let ((base-length (- (string-bytes starting-id) 2)))
             (if (and (eq (string-bytes id) (string-bytes starting-id))
                      (eq t (compare-strings starting-id 0 base-length
                                             id 0 base-length))
                      (eq (1+ left-offset) (crdt--id-offset id)))
                 (put-text-property beg end 'crdt-id starting-id-pair)
               (put-text-property beg end 'crdt-id (cons id t))))
           (crdt--split-maybe))))))
  ;; (crdt--verify-buffer)
  )

(defun crdt--local-delete (beg end)
  (let ((outer-end end))
    (crdt--with-insertion-information
     (beg 0 nil crdt--changed-string nil (length crdt--changed-string))
     (if (crdt--split-maybe)
         (let* ((not-end (< outer-end (point-max)))
                (ending-id (when not-end (crdt--get-starting-id outer-end))))
           (when (and not-end (eq starting-id (crdt--get-starting-id outer-end)))
             (crdt--set-id outer-end (crdt--id-replace-offset starting-id (+ 1 left-offset (length crdt--changed-string))))
             t))
       (crdt--with-insertion-information
        ((length crdt--changed-string) outer-end crdt--changed-string nil 0 nil)
        (crdt--split-maybe)))))
  ;; (crdt--verify-buffer)
  `(delete ,beg ,@ (crdt--dump-ids 0 (length crdt--changed-string) crdt--changed-string t)))
(defun crdt--remote-delete (message)
  (cl-destructuring-bind (position-hint . id-pairs) message
    (dolist (id-pair id-pairs)
      (cl-destructuring-bind (length . id) id-pair
        (while (> length 0)
          (goto-char (1- (crdt--find-id id position-hint)))
          (let* ((end-of-block (next-single-property-change (point) 'crdt-id nil (point-max)))
                 (block-length (- end-of-block (point))))
            (cl-case (cl-signum (- length block-length))
              ((1) (delete-char block-length)
               (cl-decf length block-length)
               (crdt--set-id-offset id (+ (crdt--id-offset id) block-length)))
              ((0) (delete-char length)
               (setq length 0))
              ((-1)
               (let* ((starting-id (crdt--get-starting-id (point)))
                      (left-offset (crdt--get-id-offset starting-id (point))))
                 (delete-char length)
                 (crdt--set-id (point) (crdt--id-replace-offset starting-id (+ left-offset length))))
               (setq length 0)))))
        ;; (crdt--verify-buffer)
        ))))

(defun crdt--before-change (beg end)
  (unless crdt--inhibit-update
    (setq crdt--changed-string (buffer-substring beg end))))

(defun crdt--after-change (beg end length)
  (when crdt--local-id ; CRDT--LOCAL-ID is NIL when a client haven't received the first sync message
    (unless crdt--inhibit-update
      (let ((crdt--inhibit-update t))
        ;; we're only interested in text change
        ;; ignore property only changes
        (save-excursion
          (goto-char beg)
          (unless (and (= length (- end beg)) (looking-at (regexp-quote crdt--changed-string)))
            (widen)
            (with-silent-modifications
              (unless (= length 0)
                (crdt--broadcast-maybe
                 (format "%S" (crdt--local-delete beg end))))
              (unless (= beg end)
                (dolist (message (crdt--local-insert beg end))
                  (crdt--broadcast-maybe
                   (format "%S" message)))))))))))

(defun crdt--remote-cursor (message)
  (unless crdt--overlay-table
    (setq crdt--overlay-table (make-hash-table)))
  (cl-destructuring-bind
      (site-id point-position-hint point-crdt-id mark-position-hint mark-crdt-id) message
    (let ((ov-pair (gethash site-id crdt--overlay-table)))
      (if point-crdt-id
          (let* ((point (crdt--find-id point-crdt-id point-position-hint))
                 (mark (if mark-crdt-id
                           (crdt--find-id mark-crdt-id mark-position-hint)
                         point)))
            (unless ov-pair
              (let ((new-cursor (make-overlay 1 1))
                    (new-region (make-overlay 1 1)))
                (overlay-put new-cursor 'face `(:background ,(crdt--get-cursor-color site-id)))
                (overlay-put new-region 'face `(:background ,(crdt--get-region-color site-id) :extend t))
                (setq ov-pair (puthash site-id (cons new-cursor new-region)
                                       crdt--overlay-table))))
            (crdt--move-cursor (car ov-pair) point)
            (crdt--move-region (cdr ov-pair) point mark))
        (when ov-pair
          (remhash site-id crdt--overlay-table)
          (delete-overlay (car ov-pair))
          (delete-overlay (cdr ov-pair)))))))

(defun crdt--local-cursor ()
  (let ((point (point))
        (mark (when (use-region-p) (mark))))
    (unless (and (eq point crdt--last-point)
                 (eq mark crdt--last-mark))
      (let ((point-id (crdt--get-id (1- point)))
            (mark-id (when mark (crdt--get-id (1- mark)))))
        `(cursor ,crdt--local-id
                 ,point ,point-id ,mark ,mark-id)))))
(defun crdt--post-command ()
  (let ((cursor-message (crdt--local-cursor)))
    (when cursor-message
      (crdt--broadcast-maybe (format "%S" cursor-message)))))


(defun crdt--dump-ids (beg end object &optional omit-end-of-block-p)
  "Serialize all CRDT ids in OBJECT from BEG to END into a list.
The list contains CONSes of the form (LENGTH CRDT-ID . END-OF-BLOCK-P),
or (LENGTH . CRDT-ID) if OMIT-END-OF-BLOCK-P is non-NIL.
in the order that they appears in the document"
  (let (ids (pos end))
    (while (> pos beg)
      (let ((prev-pos (previous-single-property-change pos 'crdt-id object beg)))
        (push (cons (- pos prev-pos)
                    (if omit-end-of-block-p
                        (crdt--get-starting-id prev-pos object)
                      (crdt--get-crdt-id-pair prev-pos object)))
              ids)
        (setq pos prev-pos)))
    ids))
(defun crdt--load-ids (ids)
  "Load the CRDT ids in IDS (generated by CRDT--DUMP-IDS)
into current buffer."
  (let ((pos (point-min)))
    (dolist (id-pair ids)
      (let ((next-pos (+ pos (car id-pair))))
        (put-text-property pos next-pos 'crdt-id (cdr id-pair))
        (setq pos next-pos)))))
(defun crdt--verify-buffer ()
  "Debug helper function.
Verify that CRDT IDs in a document follows ascending order."
  (let* ((pos (point-min))
         (id (crdt--get-starting-id pos)))
    (cl-block
        (while t
          (let* ((next-pos (next-single-property-change pos 'crdt-id))
                 (next-id (if (< next-pos (point-max))
                              (crdt--get-starting-id next-pos)
                            (cl-return)))
                 (prev-id (substring id)))
            (crdt--set-id-offset id (+ (- next-pos pos) (crdt--id-offset id)))
            (unless (string< prev-id next-id)
              (error "Not monotonic!"))
            (setq pos next-pos)
            (setq id next-id))))))

(crdt--defvar-permanent-local crdt--network-process nil)
(crdt--defvar-permanent-local crdt--network-clients nil)
(crdt--defvar-permanent-local crdt--next-client-id nil)
(cl-defun crdt--broadcast-maybe (message-string &optional (without t))
  "Broadcast or send MESSAGE-STRING.
If CRDT--NETWORK-PROCESS is a server process, broadcast MESSAGE-STRING
to clients except the one of which CLIENT-ID property is EQ to WITHOUT.
If CRDT--NETWORK-PROCESS is a server process, send MESSAGE-STRING
to server unless WITHOUT is NIL."
  ;; (message message-string)
  (if (process-contact crdt--network-process :server)
      (dolist (client crdt--network-clients)
        (when (and (eq (process-status client) 'open)
                   (not (eq (process-get client 'client-id) without)))
          (process-send-string client message-string)
          ;; (run-at-time 1 nil #'process-send-string client message-string)
          ;; ^ quick dirty way to simulate network latency
          ))
    (when without
      (process-send-string crdt--network-process message-string)
      ;; (run-at-time 1 nil #'process-send-string crdt--network-process message-string)
      )))
(defun crdt--generate-challenge ()
  (apply #'unibyte-string (cl-loop for i below 32 collect (random 256))))
(defun crdt--greet-client (process)
  (cl-pushnew process crdt--network-clients)
  (let ((client-id (process-get process 'client-id)))
    (unless client-id
      (unless (< crdt--next-client-id crdt--max-value)
        (error "Used up client IDs. Need to implement allocation algorithm."))
      (process-put process 'client-id crdt--next-client-id)
      (setq client-id crdt--next-client-id)
      (cl-incf crdt--next-client-id))
    (process-send-string process (format "%S" `(sync
                                                ,client-id
                                                ,major-mode
                                                ,(buffer-substring-no-properties (point-min) (point-max))
                                                ,@ (crdt--dump-ids (point-min) (point-max) nil))))))
(defun crdt--network-filter (process string)
  (unless (process-buffer process)
    (set-process-buffer process (generate-new-buffer "*crdt-server*"))
    (set-marker (process-mark process) 1))
  (when (buffer-live-p (process-buffer process))
    (with-current-buffer (process-buffer process)
      (when enable-multibyte-characters
        (set-buffer-multibyte nil))
      (save-excursion
        (goto-char (process-mark process))
        (insert string)
        (set-marker (process-mark process) (point))
        (goto-char (point-min))
        (let (message)
          (while (setq message (ignore-errors (read (current-buffer))))
            (with-current-buffer (process-get process 'crdt-buffer)
              (let ((server-p (process-contact crdt--network-process :server)))
                (save-excursion
                  (widen)
                  (cl-destructuring-bind (type . body) message
                    (if (or (not server-p) (process-get process 'authenticated))
                        (let ((crdt--inhibit-update t))
                          (cl-case type
                            ((insert)
                             (unless (eq (crdt--id-site (car body)) crdt--local-id)
                               (crdt--remote-insert body)
                               (crdt--broadcast-maybe (format "%S" message) (process-get process 'client-id))))
                            ((delete) (crdt--remote-delete body)
                             (crdt--broadcast-maybe (format "%S" message) (process-get process 'client-id)))
                            ((cursor) (crdt--remote-cursor body)
                             (crdt--broadcast-maybe (format "%S" message)
                                                    (process-get process 'client-id)))
                            ((sync)
                             (unless server-p ; server shouldn't receive this
                               (erase-buffer)
                               (cl-destructuring-bind (id mode content . ids) body
                                 (if (fboundp mode)
                                     (unless (eq major-mode mode)
                                       (funcall mode) ; trust your server...
                                       (crdt--install-hooks))
                                   (message "Server uses %s, but not available locally." mode))
                                 (insert content)
                                 (setq crdt--local-id id)
                                 (crdt--load-ids ids))))
                            ((challenge)
                             (unless server-p ; server shouldn't receive this
                               (message nil)
                               (let ((password (read-passwd
                                                (format "Password for %s:%s: "
                                                        (process-contact crdt--network-process :host)
                                                        (process-contact crdt--network-process :service)))))
                                 (crdt--broadcast-maybe (format "%S"
                                                                `(hello nil ,(gnutls-hash-mac 'SHA1 password (car body))))))))))
                      (cl-block nil
                        (cl-case type
                          ((hello)
                           (cl-destructuring-bind (name &optional response) body
                             (when (or (not (process-get process 'password)) ; server password is empty
                                       (and response (string-equal response (process-get process 'challenge))))
                               (process-put process 'authenticated t)
                               (crdt--greet-client process)
                               (cl-return)))))
                        (let ((challenge (crdt--generate-challenge)))
                          (process-put process 'challenge
                                       (gnutls-hash-mac 'SHA1 (substring (process-get process 'password)) challenge))
                          (process-send-string process (format "%S" `(challenge ,challenge))))))))))
            (delete-region (point-min) (point))
            (goto-char (point-min))))))))
(defun crdt--server-process-sentinel (client message)
  (with-current-buffer (process-get client 'crdt-buffer)
    (unless (eq (process-status client) 'open)
      ;; client disconnected
      (setq crdt--network-clients (delete client crdt--network-clients))
      ;; simulate a clear cursor message
      (let ((clear-cursor-message `(,(process-get client 'client-id) 1 nil 1 nil)))
        (crdt--remote-cursor clear-cursor-message)
        (crdt--broadcast-maybe (format "%S" (cons 'cursor clear-cursor-message)))))))
(defun crdt--client-process-sentinel (process message)
  (with-current-buffer (process-get process 'crdt-buffer)
    (unless (eq (process-status process) 'open)
      (crdt-stop-client))))
(defun crdt-serve-buffer (port)
  "Share the current buffer on PORT."
  (interactive "nPort: ")
  (crdt-mode)
  (setq crdt--local-id 0)
  (setq crdt--network-clients nil)
  (setq crdt--local-clock 0)
  (setq crdt--next-client-id 1)
  (save-excursion
    (widen)
    (let ((crdt--inhibit-update t))
      (crdt--local-insert (point-min) (point-max))))
  (add-hook 'kill-buffer-hook #'crdt-stop-serve-buffer nil t)
  (let ((password (read-from-minibuffer "Set password (empty for no authentication): ")))
    (setq crdt--network-process
          (make-network-process
           :name "CRDT Server"
           :server t
           :family 'ipv4
           :host "0.0.0.0"
           :service port
           :filter #'crdt--network-filter
           :sentinel #'crdt--server-process-sentinel
           :plist `(crdt-buffer ,(current-buffer) password
                                ,(when (and password (> (length password) 0)) password))))))
(defsubst crdt--clear-overlay-table ()
  (when crdt--overlay-table
    (maphash (lambda (key pair)
               (delete-overlay (car pair))
               (delete-overlay (cdr pair)))
             crdt--overlay-table)
    (setq crdt--overlay-table nil)))
(defun crdt-stop-serve-buffer ()
  "Stop sharing the current buffer."
  (interactive)
  (when crdt--network-process
    (delete-process crdt--network-process)
    (dolist (client crdt--network-clients)
      (when (process-live-p client)
        (delete-process client))
      (when (process-buffer client)
        (kill-buffer (process-buffer client))))
    (setq crdt--network-process nil)
    (setq crdt--network-clients nil)
    (crdt--clear-overlay-table)
    (setq crdt--local-id nil))
  (crdt-mode 0))
(defun crdt-stop-client ()
  "Stop the CRDT client running on current buffer if any.
Leave the buffer open."
  (interactive)
  (when crdt--network-process
    (when (process-buffer crdt--network-process)
      (kill-buffer (process-buffer crdt--network-process)))
    (delete-process crdt--network-process)
    (setq crdt--network-process nil)
    (crdt--clear-overlay-table)
    (setq crdt--local-id nil)
    (message "Disconnected from server."))
  (crdt-mode 0))
(defun crdt-connect (address port)
  "Connect to a CRDT server running at ADDRESS:PORT.
Open a new buffer to display the shared content."
  (interactive "MAddress: \nnPort: ")
  (switch-to-buffer (generate-new-buffer "CRDT Client"))
  (crdt-mode)
  (add-hook 'kill-buffer-hook #'crdt-stop-client nil t)
  (setq crdt--network-process
        (make-network-process
         :name "CRDT Client"
         :buffer (generate-new-buffer "*crdt-client*")
         :host address
         :family 'ipv4
         :service port
         :filter #'crdt--network-filter
         :sentinel #'crdt--client-process-sentinel
         :plist `(crdt-buffer ,(current-buffer))))
  (process-send-string crdt--network-process
                       (format "%S" '(hello nil)))
  (insert (format "Connected to server %s:%s, synchronizing..." address port)))
(defun crdt-test-client ()
  (interactive)
  (crdt-connect "127.0.0.1" 1333))
(defun crdt-test-server ()
  (interactive)
  (crdt-serve-buffer 1333))
(defun crdt--install-hooks ()
  (add-hook 'after-change-functions #'crdt--after-change nil t)
  (add-hook 'before-change-functions #'crdt--before-change nil t)
  (add-hook 'post-command-hook #'crdt--post-command nil t))
(defun crdt--uninstall-hooks ()
  (remove-hook 'after-change-functions #'crdt--after-change t)
  (remove-hook 'before-change-functions #'crdt--before-change t)
  (remove-hook 'post-command-hook #'crdt--post-command t))
(define-minor-mode crdt-mode
  "CRDT mode" nil " CRDT" nil
  (if crdt-mode
      (crdt--install-hooks)
    (crdt--uninstall-hooks)))

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Yuan Fu]

> On Oct 6, 2020, at 8:00 PM, Qiantan Hong <qhong@mit.edu> wrote:
> 
>> That's an interesting idea!
>> 
>> But do you intend to keep the "no server" option available too?  That is, for people whose network firewalls allow it, will it always be possible to just run crdt.el with one of the Emacsen acting as server itself?
> 
> Sure of course, that’s the easy option.
> 
> Also here is an a-bit-more-polished version.
> Now there’s
> - cursor/marked region indication
> - optional challenge-response authentication
> - I think it handles undo correctly now. Previously
> it was interfering with the text properties.
> - cleanup the dead processes etc correctly
> 
> Also for the yanking issue Jeans reported, I haven’t 
> reproduce it successfully so it is a bug it will probably 
> still be there, but you can give a try. 
> 
> <crdt.el>

Very cool! If I try to run crdt-test-client when already connected, Emacs creates a new buffer and compains "Error in post-command-hook (crdt--post-command): (wrong-type-argument stringp nil)”. Any idea why?

Yuan

Re: Question collaborative editing - Wikipedia reference

[Yuan Fu]

> On Oct 6, 2020, at 8:00 PM, Qiantan Hong <qhong@mit.edu> wrote:
> 
>> That's an interesting idea!
>> 
>> But do you intend to keep the "no server" option available too?  That is, for people whose network firewalls allow it, will it always be possible to just run crdt.el with one of the Emacsen acting as server itself?
> 
> Sure of course, that’s the easy option.
> 
> Also here is an a-bit-more-polished version.
> Now there’s
> - cursor/marked region indication
> - optional challenge-response authentication
> - I think it handles undo correctly now. Previously
> it was interfering with the text properties.
> - cleanup the dead processes etc correctly
> 
> Also for the yanking issue Jeans reported, I haven’t 
> reproduce it successfully so it is a bug it will probably 
> still be there, but you can give a try. 
> 
> <crdt.el>

Another question: According to the algorithm, what happens when both cursor are at the same place and one inserts text? Is the other cursor pushed forward or not?

Yuan

Re: Question collaborative editing - Wikipedia reference

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 860 bytes --]

> Another question: According to the algorithm, what happens when both cursor are at the same place and one inserts text? Is the other cursor pushed forward or not?
Currently the other cursor will not be pushed forward. Any suggestion for “better” behavior?
Also note that the cursor-treating part of version on this mailing list is not very polished.
I’ve made some fixes and improvements on my local version, will post shortly.
The new version will also contain Unicode support and active user list.

> Very cool! If I try to run crdt-test-client when already connected, Emacs creates a new buffer and compains "Error in post-command-hook (crdt--post-command): (wrong-type-argument stringp nil)”. Any idea why?
Thanks for the report! I haven’t consider this case. What do you think is a reasonable
behavior? Should it make a new connection?

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Yuan Fu]

> On Oct 8, 2020, at 10:12 PM, Qiantan Hong <qhong@mit.edu> wrote:
> 
>> Another question: According to the algorithm, what happens when both cursor are at the same place and one inserts text? Is the other cursor pushed forward or not?
> Currently the other cursor will not be pushed forward. Any suggestion for “better” behavior?
> Also note that the cursor-treating part of version on this mailing list is not very polished.
> I’ve made some fixes and improvements on my local version, will post shortly.
> The new version will also contain Unicode support and active user list.

Pushing forward is more intuitive IMO. I asked because I see some small problems with the cursor overlay but I’m sure you’ve fixed them now.

> 
>> Very cool! If I try to run crdt-test-client when already connected, Emacs creates a new buffer and compains "Error in post-command-hook (crdt--post-command): (wrong-type-argument stringp nil)”. Any idea why?
> Thanks for the report! I haven’t consider this case. What do you think is a reasonable
> behavior? Should it make a new connection?

It probably doesn’t make much sense to have multiple connections on a single Emacs instance to the same document. I’d just stop and complain to the user.

Yuan

Re: Question collaborative editing - Wikipedia reference

[Qiantan Hong]

[-- Attachment #1.1: Type: text/plain, Size: 281 bytes --]

Here is the current version, I feel like a relatively complete 
set of essential features is implemented now.
Changes:
- Unicode support
- fixed cursor behavior
- M-x crdt-list-users displays a list of active users. Press enter
to go to the position of that user’s cursor

[-- Attachment #1.2: crdt.el --]
[-- Type: application/octet-stream, Size: 42380 bytes --]

;;; crdt.el --- collaborative editing using Conflict-free Replicated Data Types
;;
;; Copyright (C) 2020 Qiantan Hong
;;
;; Author: Qiantan Hong <qhong@mit.edu>
;; Maintainer: Qiantan Hong <qhong@mit.edu>
;; Keywords: collaboration crdt
;;
;; crdt.el is free software: you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.
;;
;; crdt.el is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.
;;
;; You should have received a copy of the GNU General Public License
;; along with crdt.el.  If not, see <https://www.gnu.org/licenses/>.

;;; Commentary:
;; * Algorithm
;;   This packages implements the Logoot split algorithm
;;   André, Luc, et al.  "Supporting adaptable granularity of changes for massive-scale collaborative editing." 9th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing.  IEEE, 2013.
;; * Protocol
;;   Text-based version
;;   (it should be easy to migrate to a binary version.  Using text for better debugging for now)
;;   Every message takes the form (type . body)
;;   type can be: insert hello cursor challenge sync
;;   - insert
;;     body takes the form (crdt-id position-hint content)
;;     - position-hint is the buffer position where the operation happens at the site
;;       which generates the operation.  Then we can play the trick that start search
;;       near this position at other sites to speedup crdt-id search
;;     - content is the string to be inserted
;;   - delete
;;     body takes the form (position-hint (crdt-id . length)*)
;;   - cursor
;;     body takes the form
;;          (site-id point-position-hint point-crdt-id mark-position-hint mark-crdt-id)
;;     *-crdt-id can be either a CRDT ID, or
;;       - nil, which means clear the point/mark
;;   - contact
;;     body takes the form
;;          (site-id name address)
;;     when name is nil, clear the contact for this site-id
;;   - hello
;;     This message is sent from client to server, when a client connect to the server.
;;     body takes the form (client-name &optional response)
;;   - challenge
;;     body takes the form (salt)
;;   - sync
;;     This message is sent from server to client to get it sync to the state on the server.
;;     It's always sent after server receives a hello message.
;;     Might be used for error recovery or other optimization in the future.
;;     One optimization I have in mind is let server try to merge all CRDT item into a single
;;     one and try to synchronize this state to clients at best effort.
;;     body takes the form (site-id major-mode content . crdt-id-list)
;;     - site-id is the site ID the server assigned to the client
;;     - major-mode is the major mode used at the server site
;;     - content is the string in the buffer
;;     - crdt-id-list is generated from CRDT--DUMP-IDS


;;; Code:

(defgroup crdt nil
  "Collaborative editing using Conflict-free Replicated Data Types."
  :prefix "crdt-"
  :group 'applications)
(defcustom crdt-ask-for-name t
  "Ask for display name everytime a CRDT session is to be started."
  :type 'boolean)
(defcustom crdt-default-name ""
  "Default display name."
  :type 'string)
(defcustom crdt-ask-for-password t
  "Ask for server password everytime a CRDT server is to be started."
  :type 'boolean)

(require 'cl-lib)

(require 'color)
(defvar crdt-cursor-region-colors
  (let ((n 10))
    (cl-loop for i below n
             for hue by (/ 1.0 n)
             collect (cons
                      (apply #'color-rgb-to-hex
                             (color-hsl-to-rgb hue 0.5 0.5))
                      (apply #'color-rgb-to-hex
                             (color-hsl-to-rgb hue 0.2 0.5))))))

(defun crdt--get-cursor-color (site-id)
  "Get cursor color for SITE-ID."
  (car (nth (mod site-id (length crdt-cursor-region-colors)) crdt-cursor-region-colors)))
(defun crdt--get-region-color (site-id)
  "Get region color for SITE-ID."
  (cdr (nth (mod site-id (length crdt-cursor-region-colors)) crdt-cursor-region-colors)))
(defun crdt--move-cursor (ov pos)
  "Move pseudo cursor overlay OV to POS."
  ;; Hax!
  (let* ((eof (eq pos (point-max)))
         (eol (unless eof (eq (char-after pos) ?\n)))
         (end (if eof pos (1+ pos)))
         (display-string
          (when eof
            (unless (or (eq (point) (point-max))
                        (cl-some (lambda (ov)
                                   (and (eq (overlay-get ov 'category) 'crdt-pseudo-cursor)
                                        (overlay-get ov 'before-string)))
                                 (overlays-in (point-max) (point-max))))
              (propertize " " 'face (overlay-get ov 'face))))))
    (move-overlay ov pos end)
    (overlay-put ov 'before-string display-string)))
(defun crdt--move-region (ov pos mark)
  "Move pseudo marked region overlay OV to mark between POS and MARK."
  (move-overlay ov (min pos mark) (max pos mark)))


;; CRDT IDs are represented by unibyte strings (for efficient comparison)
;; Every two bytes represent a big endian encoded integer
;; For base IDs, last two bytes are always representing site ID
;; Stored strings are BASE-ID:OFFSETs. So the last two bytes represent offset,
;; and second last two bytes represent site ID
(defconst crdt--max-value (lsh 1 16))
;; (defconst crdt--max-value 4)
;; for debug
(defconst crdt--low-byte-mask 255)
(defsubst crdt--get-two-bytes (string index)
  "Get the big-endian encoded integer from STRING starting from INDEX.
INDEX is counted by bytes."
  (logior (lsh (elt string index) 8)
          (elt string (1+ index))))
(defsubst crdt--get-two-bytes-with-offset (string offset index default)
  "Helper function for CRDT--GENERATE-ID.
Get the big-endian encoded integer from STRING starting from INDEX,
but with last two-bytes of STRING (the offset portion) replaced by OFFSET,
and padded infintely by DEFAULT to the right."
  (cond ((= index (- (string-bytes string) 2))
         offset)
        ((< (1+ index) (string-bytes string))
         (logior (lsh (elt string index) 8)
                 (elt string (1+ index))))
        (t default)))

(defsubst crdt--id-offset (id)
  "Get the literal offset integer from ID.
Note that it might deviate from real offset for a character
in the middle of a block."
  (crdt--get-two-bytes id (- (string-bytes id) 2)))
(defsubst crdt--set-id-offset (id offset)
  "Set the OFFSET portion of ID destructively."
  (let ((length (string-bytes id)))
    (aset id (- length 2) (lsh offset -8))
    (aset id (- length 1) (logand offset crdt--low-byte-mask))))
(defsubst crdt--id-replace-offset (id offset)
  "Create and return a new id string by replacing the OFFSET portion from ID."
  (let ((new-id (substring id)))
    (crdt--set-id-offset new-id offset)
    new-id))
(defsubst crdt--id-site (id)
  "Get the site id from ID."
  (crdt--get-two-bytes id (- (string-bytes id) 4)))
(defsubst crdt--generate-id (low-id low-offset high-id high-offset site-id)
  "Generate a new ID between LOW-ID and HIGH-ID.
The generating site is marked as SITE-ID.
Offset parts of LOW-ID and HIGH-ID are overriden by LOW-OFFSET
and HIGH-OFFSET.  (to save two copying from using CRDT--ID-REPLACE-OFFSET)"
  (let* ((l (crdt--get-two-bytes-with-offset low-id low-offset 0 0))
         (h (crdt--get-two-bytes-with-offset high-id high-offset 0 crdt--max-value))
         (bytes (cl-loop for pos from 2 by 2
                         while (< (- h l) 2)
                         append (list (lsh l -8)
                                      (logand l crdt--low-byte-mask))
                         do (setq l (crdt--get-two-bytes-with-offset low-id low-offset pos 0))
                         do (setq h (crdt--get-two-bytes-with-offset high-id high-offset pos crdt--max-value))))
         (m (+ l 1 (random (- h l 1)))))
    (apply #'unibyte-string
           (append bytes (list (lsh m -8)
                               (logand m crdt--low-byte-mask)
                               (lsh site-id -8)
                               (logand site-id crdt--low-byte-mask)
                               0
                               0)))))

;; CRDT-ID text property actually stores a cons of (ID-STRING . END-OF-BLOCK-P)
(defsubst crdt--get-crdt-id-pair (pos &optional obj)
  "Get the (CRDT-ID . END-OF-BLOCK-P) pair at POS in OBJ."
  (get-text-property pos 'crdt-id obj))
(defsubst crdt--get-starting-id (pos &optional obj)
  "Get the CRDT-ID at POS in OBJ."
  (car (crdt--get-crdt-id-pair pos obj)))
(defsubst crdt--end-of-block-p (pos &optional obj)
  "Get the END-OF-BLOCK-P at POS in OBJ."
  (cdr (crdt--get-crdt-id-pair pos obj)))
(defsubst crdt--get-starting-id-maybe (pos &optional obj limit)
  "Get the CRDT-ID at POS in OBJ if POS is no smaller than LIMIT.
Return NIL otherwise."
  (unless (< pos (or limit (point-min)))
    (car (get-text-property pos 'crdt-id obj))))
(defsubst crdt--get-id-offset (starting-id pos &optional obj limit)
  "Get the real offset integer for a character at POS.
Assume the stored literal ID is STARTING-ID."
  (let* ((start-pos (previous-single-property-change (1+ pos) 'crdt-id obj (or limit (point-min)))))
    (+ (- pos start-pos) (crdt--id-offset starting-id))))
(defsubst crdt--get-id (pos &optional obj limit)
  "Get the real CRDT ID at POS."
  (let ((limit (or limit (point-min))))
    (if (< pos limit) ""
      (let* ((starting-id (crdt--get-starting-id pos obj))
             (left-offset (crdt--get-id-offset starting-id pos obj limit)))
        (crdt--id-replace-offset starting-id left-offset)))))

(defsubst crdt--set-id (pos id &optional end-of-block-p obj limit)
  "Set the crdt ID and END-OF-BLOCK-P at POS in OBJ.
Any characters after POS but before LIMIT that used to
have the same (CRDT-ID . END-OF-BLOCK-P) pair are also updated
with ID and END-OF-BLOCK-P."
  (put-text-property pos (next-single-property-change pos 'crdt-id obj (or limit (point-max))) 'crdt-id (cons id end-of-block-p) obj))

(cl-defmacro crdt--with-insertion-information
    ((beg end &optional beg-obj end-obj beg-limit end-limit) &body body)
  "Helper macro to setup some useful variables."
  `(let* ((not-begin (> ,beg ,(or beg-limit '(point-min)))) ; if it's nil, we're at the beginning of buffer
          (left-pos (1- ,beg))
          (starting-id-pair (when not-begin (crdt--get-crdt-id-pair left-pos ,beg-obj)))
          (starting-id (if not-begin (car starting-id-pair) ""))
          (left-offset (if not-begin (crdt--get-id-offset starting-id left-pos ,beg-obj ,beg-limit) 0))
          (not-end (< ,end ,(or end-limit '(point-max))))
          (ending-id (if not-end (crdt--get-starting-id ,end ,end-obj) ""))
          (right-offset (if not-end (crdt--id-offset ending-id) 0))
          (beg ,beg)
          (end ,end)
          (beg-obj ,beg-obj)
          (end-obj ,end-obj)
          (beg-limit ,beg-limit)
          (end-limit ,end-limit))
     ,@body))
(defmacro crdt--split-maybe ()
  '(when (and not-end (eq starting-id (crdt--get-starting-id end end-obj)))
     ;; need to split id block
     (crdt--set-id end (crdt--id-replace-offset starting-id (1+ left-offset))
                   (crdt--end-of-block-p left-pos beg-obj) end-obj end-limit)
     (rplacd (get-text-property left-pos 'crdt-id beg-obj) nil) ;; clear end-of-block flag
     t))

(defsubst crdt--same-base-p (a b)
  (let* ((a-length (string-bytes a))
         (b-length (string-bytes b)))
    (and (eq a-length b-length)
         (let ((base-length (- a-length 2)))
           (eq t (compare-strings a 0 base-length b 0 base-length))))))

(defmacro crdt--defvar-permanent-local (name &optional val docstring)
  `(progn
     (defvar-local ,name ,val ,docstring)
     (put ',name 'permanent-local t)))
(crdt--defvar-permanent-local crdt--local-id nil "Local site-id.")
(crdt--defvar-permanent-local crdt--inhibit-update nil "When set, don't call CRDT--LOCAL-* on change.
This is useful for functions that apply remote change to local buffer,
to avoid recusive calling of CRDT synchronization functions.")
(crdt--defvar-permanent-local crdt--changed-string nil)
(crdt--defvar-permanent-local crdt--last-point nil)
(crdt--defvar-permanent-local crdt--last-mark nil)
(crdt--defvar-permanent-local crdt--overlay-table nil
                              "A hash table that maps SITE-ID to CONSes of the form (CURSOR-OVERLAY . REGION-OVERLAY).")
(crdt--defvar-permanent-local crdt--contact-table nil
                              "A hash table that maps SITE-ID to LISTs of the form (DISPLAY-NAME ADDRESS).")
(crdt--defvar-permanent-local crdt--local-name nil)
(crdt--defvar-permanent-local crdt--user-list-buffer nil)

(defvar crdt-user-list-mode-map
  (let ((map (make-sparse-keymap)))
    (define-key map (kbd "RET") #'crdt-user-list-goto)
    map))
(define-derived-mode crdt-user-list-mode tabulated-list-mode
  "CRDT User List"
  (setq tabulated-list-format [("Display name" 15 t)
                               ("Address" 15 t)
                               ("Port" 7 t)]))
(defun crdt-user-list-goto ()
  (interactive)
  (let ((site-id (tabulated-list-get-id)))
    (switch-to-buffer-other-window crdt--user-list-parent)
    (when site-id
      (goto-char (overlay-start (car (gethash site-id crdt--overlay-table)))))))
(crdt--defvar-permanent-local crdt--user-list-parent nil "Set to the CRDT shared buffer, local in a CRDT-USER-LIST buffer.")
(defun crdt-list-users (&optional crdt-buffer display-buffer)
  "Display a list of active users working on a CRDT-shared buffer CRDT-BUFFER.
If DISPLAY-BUFFER is provided, display the output there. Otherwise use a dedicated
buffer for displaying active users on CRDT-BUFFER."
  (interactive)
  (with-current-buffer (or crdt-buffer (current-buffer))
    (unless crdt-mode
      (error "Not a CRDT shared buffer."))
    (unless display-buffer
      (unless (and crdt--user-list-buffer (buffer-live-p crdt--user-list-buffer))
        (let ((crdt-buffer (current-buffer)))
          (setq crdt--user-list-buffer
                (generate-new-buffer (concat (buffer-name (current-buffer))
                                             " users")))
          (with-current-buffer crdt--user-list-buffer
            (setq crdt--user-list-parent crdt-buffer))))
      (setq display-buffer crdt--user-list-buffer))
    (crdt-refresh-users display-buffer)
    (switch-to-buffer-other-window display-buffer)))
(defun crdt-refresh-users (display-buffer)
  (let ((table crdt--contact-table)
        (local-name crdt--local-name))
    (with-current-buffer display-buffer
      (crdt-user-list-mode)
      (setq tabulated-list-entries nil)
      (push (list crdt--local-id (vector local-name "*myself*" "--")) tabulated-list-entries)
      (maphash (lambda (k v)
                 (push (list k (cl-destructuring-bind (name contact) v
                                 (let ((colored-name (concat name " ")))
                                   (put-text-property 0 (1- (length colored-name))
                                                      'face `(:background ,(crdt--get-region-color k))
                                                      colored-name)
                                   (put-text-property (1- (length colored-name)) (length colored-name)
                                                      'face `(:background ,(crdt--get-cursor-color k))
                                                      colored-name)
                                   (vector colored-name (car contact) (format "%s" (cadr contact))))))
                       tabulated-list-entries))
               table)
      (tabulated-list-init-header)
      (tabulated-list-print))))
(defsubst crdt--refresh-users-maybe ()
  (when (and crdt--user-list-buffer (buffer-live-p crdt--user-list-buffer))
    (crdt-refresh-users crdt--user-list-buffer)))

(defun crdt--local-insert (beg end)
  "To be called after a local insert happened in current buffer from BEG to END.
Returns a list of (insert type) messages to be sent."
  (let (resulting-commands)
    (crdt--with-insertion-information
     (beg end)
     (unless (crdt--split-maybe)
       (when (and not-begin
                  (eq (crdt--id-site starting-id) crdt--local-id)
                  (crdt--end-of-block-p left-pos))
         ;; merge crdt id block
         (let* ((max-offset crdt--max-value)
                (merge-end (min end (+ (- max-offset left-offset 1) beg))))
           (unless (= merge-end beg)
             (put-text-property beg merge-end 'crdt-id starting-id-pair)
             (let ((virtual-id (substring starting-id)))
               (crdt--set-id-offset virtual-id (1+ left-offset))
               (push `(insert ,(base64-encode-string virtual-id) ,beg
                              ,(buffer-substring-no-properties beg merge-end))
                     resulting-commands))
             (setq beg merge-end)))))
     (while (< beg end)
       (let ((block-end (min end (+ crdt--max-value beg))))
         (let ((new-id (crdt--generate-id starting-id left-offset ending-id right-offset crdt--local-id)))
           (put-text-property beg block-end 'crdt-id (cons new-id t))
           (push `(insert ,(base64-encode-string new-id) ,beg
                          ,(buffer-substring-no-properties beg block-end))
                 resulting-commands)
           (setq beg block-end)
           (setq left-offset (1- crdt--max-value)) ; this is always true when we need to continue
           (setq starting-id new-id)))))
    ;; (crdt--verify-buffer)
    (nreverse resulting-commands)))

(defun crdt--find-id (id pos)
  "Find the first position *after* ID.  Start the search from POS."
  (let* ((left-pos (previous-single-property-change (if (< pos (point-max)) (1+ pos) pos)
                                                    'crdt-id nil (point-min)))
         (left-id (crdt--get-starting-id left-pos))
         (right-pos (next-single-property-change pos 'crdt-id nil (point-max)))
         (right-id (crdt--get-starting-id right-pos)))
    (cl-block nil
      (while t
        (cond ((<= right-pos (point-min))
               (cl-return (point-min)))
              ((>= left-pos (point-max))
               (cl-return (point-max)))
              ((and right-id (not (string< id right-id)))
               (setq left-pos right-pos)
               (setq left-id right-id)
               (setq right-pos (next-single-property-change right-pos 'crdt-id nil (point-max)))
               (setq right-id (crdt--get-starting-id right-pos)))
              ((string< id left-id)
               (setq right-pos left-pos)
               (setq right-id left-id)
               (setq left-pos (previous-single-property-change left-pos 'crdt-id nil (point-min)))
               (setq left-id (crdt--get-starting-id left-pos)))
              (t
               ;; will unibyte to multibyte conversion cause any problem?
               (cl-return
                (if (eq t (compare-strings left-id 0 (- (string-bytes left-id) 2)
                                           id 0 (- (string-bytes left-id) 2)))
                    (min right-pos (+ left-pos 1
                                      (- (crdt--get-two-bytes id (- (string-bytes left-id) 2))
                                         (crdt--id-offset left-id))))
                  right-pos))))))))
(defun crdt--remote-insert (message)
  (let ((crdt--inhibit-update t))
    (cl-destructuring-bind (id-base64 position-hint content) message
      (let* ((id (base64-decode-string id-base64))
             (beg (crdt--find-id id position-hint)) end)
        (goto-char beg)
        (insert content)
        (setq end (point))
        (with-silent-modifications
          (crdt--with-insertion-information
           (beg end)
           (let ((base-length (- (string-bytes starting-id) 2)))
             (if (and (eq (string-bytes id) (string-bytes starting-id))
                      (eq t (compare-strings starting-id 0 base-length
                                             id 0 base-length))
                      (eq (1+ left-offset) (crdt--id-offset id)))
                 (put-text-property beg end 'crdt-id starting-id-pair)
               (put-text-property beg end 'crdt-id (cons id t))))
           (crdt--split-maybe))))))
  ;; (crdt--verify-buffer)
  )

(defun crdt--local-delete (beg end)
  (let ((outer-end end))
    (crdt--with-insertion-information
     (beg 0 nil crdt--changed-string nil (length crdt--changed-string))
     (if (crdt--split-maybe)
         (let* ((not-end (< outer-end (point-max)))
                (ending-id (when not-end (crdt--get-starting-id outer-end))))
           (when (and not-end (eq starting-id (crdt--get-starting-id outer-end)))
             (crdt--set-id outer-end (crdt--id-replace-offset starting-id (+ 1 left-offset (length crdt--changed-string))))
             t))
       (crdt--with-insertion-information
        ((length crdt--changed-string) outer-end crdt--changed-string nil 0 nil)
        (crdt--split-maybe)))))
  ;; (crdt--verify-buffer)
  `(delete ,beg ,@ (crdt--dump-ids 0 (length crdt--changed-string) crdt--changed-string t)))
(defun crdt--remote-delete (message)
  (cl-destructuring-bind (position-hint . id-pairs) message
    (dolist (id-pair id-pairs)
      (cl-destructuring-bind (length . id-base64) id-pair
        (let ((id (base64-decode-string id-base64)))
          (while (> length 0)
            (goto-char (1- (crdt--find-id id position-hint)))
            (let* ((end-of-block (next-single-property-change (point) 'crdt-id nil (point-max)))
                   (block-length (- end-of-block (point))))
              (cl-case (cl-signum (- length block-length))
                ((1) (delete-char block-length)
                 (cl-decf length block-length)
                 (crdt--set-id-offset id (+ (crdt--id-offset id) block-length)))
                ((0) (delete-char length)
                 (setq length 0))
                ((-1)
                 (let* ((starting-id (crdt--get-starting-id (point)))
                        (left-offset (crdt--get-id-offset starting-id (point))))
                   (delete-char length)
                   (crdt--set-id (point) (crdt--id-replace-offset starting-id (+ left-offset length))))
                 (setq length 0))))))
        ;; (crdt--verify-buffer)
        ))))

(defun crdt--before-change (beg end)
  (unless crdt--inhibit-update
    (setq crdt--changed-string (buffer-substring beg end))))

(defun crdt--after-change (beg end length)
  (mapc (lambda (ov)
          (when (eq (overlay-get ov 'category) 'crdt-pseudo-cursor)
            (crdt--move-cursor ov beg)))
        (overlays-in beg (min (point-max) (1+ beg))))
  (when crdt--local-id ; CRDT--LOCAL-ID is NIL when a client haven't received the first sync message
    (unless crdt--inhibit-update
      (let ((crdt--inhibit-update t))
        ;; we're only interested in text change
        ;; ignore property only changes
        (save-excursion
          (goto-char beg)
          (unless (and (= length (- end beg)) (looking-at (regexp-quote crdt--changed-string)))
            (widen)
            (with-silent-modifications
              (unless (= length 0)
                (crdt--broadcast-maybe
                 (format "%S" (crdt--local-delete beg end))))
              (unless (= beg end)
                (dolist (message (crdt--local-insert beg end))
                  (crdt--broadcast-maybe
                   (format "%S" message)))))))))))

(defun crdt--remote-cursor (message)
  (cl-destructuring-bind
      (site-id point-position-hint point-crdt-id-base64 mark-position-hint mark-crdt-id-base64) message
    (let ((ov-pair (gethash site-id crdt--overlay-table)))
      (if point-crdt-id-base64
          (let* ((point (crdt--find-id (base64-decode-string point-crdt-id-base64) point-position-hint))
                 (mark (if mark-crdt-id-base64
                           (crdt--find-id (base64-decode-string mark-crdt-id-base64) mark-position-hint)
                         point)))
            (unless ov-pair
              (let ((new-cursor (make-overlay 1 1))
                    (new-region (make-overlay 1 1)))
                (overlay-put new-cursor 'face `(:background ,(crdt--get-cursor-color site-id)))
                (overlay-put new-cursor 'category 'crdt-pseudo-cursor)
                (overlay-put new-region 'face `(:background ,(crdt--get-region-color site-id) :extend t))
                (setq ov-pair (puthash site-id (cons new-cursor new-region)
                                       crdt--overlay-table))))
            (crdt--move-cursor (car ov-pair) point)
            (crdt--move-region (cdr ov-pair) point mark))
        (when ov-pair
          (remhash site-id crdt--overlay-table)
          (delete-overlay (car ov-pair))
          (delete-overlay (cdr ov-pair)))))))

(cl-defun crdt--local-cursor (&optional (lazy t))
  (let ((point (point))
        (mark (when (use-region-p) (mark))))
    (unless (and lazy
                 (eq point crdt--last-point)
                 (eq mark crdt--last-mark))
      (when (or (eq point (point-max)) (eq crdt--last-point (point-max)))
        (mapc (lambda (ov)
                (when (eq (overlay-get ov 'category) 'crdt-pseudo-cursor)
                  (crdt--move-cursor ov (point-max))))
              (overlays-in (point-max) (point-max))))
      (setq crdt--last-point point)
      (setq crdt--last-mark mark)
      (let ((point-id-base64 (base64-encode-string (crdt--get-id (1- point))))
            (mark-id-base64 (when mark (base64-encode-string (crdt--get-id (1- mark))))))
        `(cursor ,crdt--local-id
                 ,point ,point-id-base64 ,mark ,mark-id-base64)))))
(defun crdt--post-command ()
  (let ((cursor-message (crdt--local-cursor)))
    (when cursor-message
      (crdt--broadcast-maybe (format "%S" cursor-message)))))


(defun crdt--dump-ids (beg end object &optional omit-end-of-block-p)
  "Serialize all CRDT ids in OBJECT from BEG to END into a list.
The list contains CONSes of the form (LENGTH CRDT-ID-BASE64 . END-OF-BLOCK-P),
or (LENGTH . CRDT-ID-BASE64) if OMIT-END-OF-BLOCK-P is non-NIL.
in the order that they appears in the document"
  (let (ids (pos end))
    (while (> pos beg)
      (let ((prev-pos (previous-single-property-change pos 'crdt-id object beg)))
        (push (cons (- pos prev-pos)
                    (cl-destructuring-bind (id . eob) (crdt--get-crdt-id-pair prev-pos object)
                      (let ((id-base64 (base64-encode-string id)))
                        (if omit-end-of-block-p id-base64 (cons id-base64 eob)))))
              ids)
        (setq pos prev-pos)))
    ids))
(defun crdt--load-ids (ids)
  "Load the CRDT ids in IDS (generated by CRDT--DUMP-IDS)
into current buffer."
  (let ((pos (point-min)))
    (dolist (id-pair ids)
      (let ((next-pos (+ pos (car id-pair))))
        (put-text-property pos next-pos 'crdt-id
                           (cons (base64-decode-string (cadr id-pair)) (cddr id-pair)))
        (setq pos next-pos)))))
(defun crdt--verify-buffer ()
  "Debug helper function.
Verify that CRDT IDs in a document follows ascending order."
  (let* ((pos (point-min))
         (id (crdt--get-starting-id pos)))
    (cl-block
        (while t
          (let* ((next-pos (next-single-property-change pos 'crdt-id))
                 (next-id (if (< next-pos (point-max))
                              (crdt--get-starting-id next-pos)
                            (cl-return)))
                 (prev-id (substring id)))
            (crdt--set-id-offset id (+ (- next-pos pos) (crdt--id-offset id)))
            (unless (string< prev-id next-id)
              (error "Not monotonic!"))
            (setq pos next-pos)
            (setq id next-id))))))

(crdt--defvar-permanent-local crdt--network-process nil)
(crdt--defvar-permanent-local crdt--network-clients nil)
(crdt--defvar-permanent-local crdt--next-client-id nil)
(cl-defun crdt--broadcast-maybe (message-string &optional (without t))
  "Broadcast or send MESSAGE-STRING.
If CRDT--NETWORK-PROCESS is a server process, broadcast MESSAGE-STRING
to clients except the one of which CLIENT-ID property is EQ to WITHOUT.
If CRDT--NETWORK-PROCESS is a server process, send MESSAGE-STRING
to server unless WITHOUT is NIL."
  ;; (message message-string)
  (if (process-contact crdt--network-process :server)
      (dolist (client crdt--network-clients)
        (when (and (eq (process-status client) 'open)
                   (not (eq (process-get client 'client-id) without)))
          (process-send-string client message-string)
          ;; (run-at-time 1 nil #'process-send-string client message-string)
          ;; ^ quick dirty way to simulate network latency, for debugging
          ))
    (when without
      (process-send-string crdt--network-process message-string)
      ;; (run-at-time 1 nil #'process-send-string crdt--network-process message-string)
      )))
(defun crdt--generate-challenge ()
  (apply #'unibyte-string (cl-loop for i below 32 collect (random 256))))
(defun crdt--greet-client (process)
  (cl-pushnew process crdt--network-clients)
  (let ((client-id (process-get process 'client-id)))
    (unless client-id
      (unless (< crdt--next-client-id crdt--max-value)
        (error "Used up client IDs. Need to implement allocation algorithm."))
      (process-put process 'client-id crdt--next-client-id)
      (setq client-id crdt--next-client-id)
      (cl-incf crdt--next-client-id))
    (process-send-string process (format "%S" `(sync
                                                ,client-id
                                                ,major-mode
                                                ,(buffer-substring-no-properties (point-min) (point-max))
                                                ,@ (crdt--dump-ids (point-min) (point-max) nil))))
    (maphash (lambda (site-id ov-pair)
               (cl-destructuring-bind (cursor-ov . region-ov) ov-pair
                 (let* ((point (overlay-start cursor-ov))
                        (region-beg (overlay-start region-ov))
                        (region-end (overlay-end region-ov))
                        (mark (if (eq point region-beg)
                                  (unless (eq point region-end) region-end)
                                region-beg))
                        (point-id-base64 (base64-encode-string (crdt--get-id (1- point))))
                        (mark-id-base64 (when mark (base64-encode-string (crdt--get-id (1- mark))))))
                   (process-send-string process
                                        (format "%S"
                                                `(cursor ,site-id
                                                         ,point ,point-id-base64 ,mark ,mark-id-base64))))))
             crdt--overlay-table)
    (process-send-string process (format "%S" (crdt--local-cursor nil)))
    (maphash (lambda (k v)
               (process-send-string process (format "%S" `(contact ,k ,@v))))
             crdt--contact-table)
    (process-send-string process
                         (format "%S" `(contact ,crdt--local-id
                                                ,crdt--local-name nil)))
    (let ((contact-message `(contact ,client-id ,(process-get process 'client-name)
                                     ,(process-contact process))))
      (crdt--broadcast-maybe (format "%S" contact-message) client-id)
      (crdt-process-message contact-message nil))))

(cl-defgeneric crdt-process-message (message process))
(cl-defmethod crdt-process-message ((message (head insert)) process)
  (crdt--remote-insert (cdr message))
  (crdt--broadcast-maybe (format "%S" message) (process-get process 'client-id)))
(cl-defmethod crdt-process-message ((message (head delete)) process)
  (crdt--remote-delete (cdr message))
  (crdt--broadcast-maybe (format "%S" message) (process-get process 'client-id)))
(cl-defmethod crdt-process-message ((message (head cursor)) process)
  (crdt--remote-cursor (cdr message))
  (crdt--broadcast-maybe (format "%S" message) (process-get process 'client-id)))
(cl-defmethod crdt-process-message ((message (head sync)) process)
  (unless (crdt--server-p) ; server shouldn't receive this
    (erase-buffer)
    (cl-destructuring-bind (id mode content . ids) (cdr message)
      (if (fboundp mode)
          (unless (eq major-mode mode)
            (funcall mode) ; trust your server...
            (crdt-mode))
        (message "Server uses %s, but not available locally." mode))
      (insert content)
      (setq crdt--local-id id)
      (crdt--load-ids ids)
      (puthash 0 (list nil (process-contact process)) crdt--contact-table))))
(cl-defmethod crdt-process-message ((message (head challenge)) process)
  (unless (crdt--server-p) ; server shouldn't receive this
    (message nil)
    (let ((password (read-passwd
                     (format "Password for %s:%s: "
                             (process-contact crdt--network-process :host)
                             (process-contact crdt--network-process :service)))))
      (crdt--broadcast-maybe (format "%S"
                                     `(hello nil ,(gnutls-hash-mac 'SHA1 password (cadr message))))))))
(cl-defmethod crdt-process-message ((message (head contact)) process)
  (cl-destructuring-bind (site-id display-name address) (cdr message)
    (if display-name
        (puthash site-id (list display-name
                               (or address (cadr (gethash site-id crdt--contact-table))))
                 crdt--contact-table)
      (remhash site-id crdt--contact-table))
    (crdt--refresh-users-maybe)))

(defsubst crdt--server-p ()
  (process-contact crdt--network-process :server))

(defun crdt--network-filter (process string)
  (unless (process-buffer process)
    (set-process-buffer process (generate-new-buffer "*crdt-server*"))
    (set-marker (process-mark process) 1))
  (when (buffer-live-p (process-buffer process))
    (with-current-buffer (process-buffer process)
      (save-excursion
        (goto-char (process-mark process))
        (insert string)
        (set-marker (process-mark process) (point))
        (goto-char (point-min))
        (let (message)
          (while (setq message (ignore-errors (read (current-buffer))))
            ;; (print message)
            (with-current-buffer (process-get process 'crdt-buffer)
              (save-excursion
                (widen)
                (if (or (not (crdt--server-p)) (process-get process 'authenticated))
                    (let ((crdt--inhibit-update t))
                      (crdt-process-message message process))
                  (cl-block nil
                    (when (eq (car message) 'hello)
                      (cl-destructuring-bind (name &optional response) (cdr message)
                        (when (or (not (process-get process 'password)) ; server password is empty
                                  (and response (string-equal response (process-get process 'challenge))))
                          (process-put process 'authenticated t)
                          (process-put process 'client-name name)
                          (crdt--greet-client process)
                          (cl-return))))
                    (let ((challenge (crdt--generate-challenge)))
                      (process-put process 'challenge
                                   (gnutls-hash-mac 'SHA1 (substring (process-get process 'password)) challenge))
                      (process-send-string process (format "%S" `(challenge ,challenge))))))))
            (delete-region (point-min) (point))
            (goto-char (point-min))))))))
(defun crdt--server-process-sentinel (client message)
  (with-current-buffer (process-get client 'crdt-buffer)
    (unless (eq (process-status client) 'open)
      ;; client disconnected
      (setq crdt--network-clients (delete client crdt--network-clients))
      ;; generate a clear cursor message and a clear contact message
      (let* ((client-id (process-get client 'client-id))
             (clear-cursor-message `(cursor ,client-id 1 nil 1 nil))
             (clear-contact-message `(contact ,client-id nil nil)))
        (crdt-process-message clear-cursor-message client)
        (crdt-process-message clear-contact-message client)
        (crdt-refresh-users-maybe)))))
(defun crdt--client-process-sentinel (process message)
  (with-current-buffer (process-get process 'crdt-buffer)
    (unless (eq (process-status process) 'open)
      (crdt-stop-client))))
(defun crdt-serve-buffer (port &optional password name)
  "Share the current buffer on PORT."
  (interactive "nPort: ")
  (crdt-mode)
  (setq crdt--local-id 0)
  (setq crdt--network-clients nil)
  (setq crdt--local-clock 0)
  (setq crdt--next-client-id 1)
  (save-excursion
    (widen)
    (let ((crdt--inhibit-update t))
      (with-silent-modifications
        (crdt--local-insert (point-min) (point-max)))))
  (add-hook 'kill-buffer-hook #'crdt-stop-serve-buffer nil t)
  (unless password
    (setq password
          (when crdt-ask-for-password
            (read-from-minibuffer "Set password (empty for no authentication): "))))
  (unless name
    (when crdt-ask-for-name
      (setq name (read-from-minibuffer "Display name: "))))
  (setq crdt--local-name name)
  (setq crdt--network-process
        (make-network-process
         :name "CRDT Server"
         :server t
         :family 'ipv4
         :host "0.0.0.0"
         :service port
         :filter #'crdt--network-filter
         :sentinel #'crdt--server-process-sentinel
         :plist `(crdt-buffer ,(current-buffer) password
                              ,(when (and password (> (length password) 0)) password)))))
(defsubst crdt--clear-overlay-table ()
  (when crdt--overlay-table
    (maphash (lambda (key pair)
               (delete-overlay (car pair))
               (delete-overlay (cdr pair)))
             crdt--overlay-table)
    (setq crdt--overlay-table nil)))
(defun crdt-stop-serve-buffer ()
  "Stop sharing the current buffer."
  (interactive)
  (if (or (not crdt--network-process)
          (not (process-contact crdt--network-process :server)))
      (message "No CRDT server running on current buffer.")
    (when (process-buffer crdt--network-process)
      (kill-buffer (process-buffer crdt--network-process)))
    (delete-process crdt--network-process)
    (dolist (client crdt--network-clients)
      (when (process-live-p client)
        (delete-process client))
      (when (process-buffer client)
        (kill-buffer (process-buffer client))))
    (setq crdt--network-process nil)
    (setq crdt--network-clients nil)
    (crdt--clear-overlay-table)
    (setq crdt--local-id nil)
    (setq crdt--contact-table nil))
  (crdt-mode 0))
(defun crdt-stop-client ()
  "Stop the CRDT client running on current buffer if any.
Leave the buffer open."
  (interactive)
  (if (or (not crdt--network-process) (process-contact crdt--network-process :server))
      (message "No CRDT client running on current buffer.")
    (when (process-buffer crdt--network-process)
      (kill-buffer (process-buffer crdt--network-process)))
    (delete-process crdt--network-process)
    (setq crdt--network-process nil)
    (crdt--clear-overlay-table)
    (setq crdt--local-id nil)
    (setq crdt--contact-table nil)
    (message "Disconnected from server."))
  (crdt-mode 0))
(defun crdt-connect (address port &optional name)
  "Connect to a CRDT server running at ADDRESS:PORT.
Open a new buffer to display the shared content."
  (interactive "MAddress: \nnPort: ")
  (switch-to-buffer (generate-new-buffer "CRDT Client"))
  (unless name
    (when crdt-ask-for-name
      (setq name (read-from-minibuffer "Display name: "))))
  (setq crdt--local-name name)
  (setq crdt--network-process
        (make-network-process
         :name "CRDT Client"
         :buffer (generate-new-buffer "*crdt-client*")
         :host address
         :family 'ipv4
         :service port
         :filter #'crdt--network-filter
         :sentinel #'crdt--client-process-sentinel
         :plist `(crdt-buffer ,(current-buffer))))
  (crdt-mode)
  (add-hook 'kill-buffer-hook #'crdt-stop-client nil t)
  (process-send-string crdt--network-process
                       (format "%S" `(hello ,name)))
  (insert (format "Connected to server %s:%s, synchronizing..." address port)))
(defun crdt-test-client ()
  (interactive)
  (crdt-connect "127.0.0.1" 1333))
(defun crdt-test-server ()
  (interactive)
  (crdt-serve-buffer 1333))
(defun crdt--install-hooks ()
  (add-hook 'after-change-functions #'crdt--after-change nil t)
  (add-hook 'before-change-functions #'crdt--before-change nil t)
  (add-hook 'post-command-hook #'crdt--post-command nil t))
(defun crdt--uninstall-hooks ()
  (remove-hook 'after-change-functions #'crdt--after-change t)
  (remove-hook 'before-change-functions #'crdt--before-change t)
  (remove-hook 'post-command-hook #'crdt--post-command t))
(define-minor-mode crdt-mode
  "CRDT mode" nil " CRDT" nil
  (if crdt-mode
      (progn
        (setq crdt--overlay-table (make-hash-table))
        (setq crdt--contact-table (make-hash-table))
        (crdt--install-hooks))
    (crdt--uninstall-hooks)
    (when crdt--user-list-buffer
      (kill-buffer crdt--user-list-buffer)
      (setq crdt--user-list-buffer nil))))

(provide 'crdt)

[-- Attachment #1.3: Type: text/plain, Size: 1930 bytes --]


It seems that locally I didn’t reproduce yanking issue and multiple 
crdt-test-client issue — it just open another same document normally,
which is not the best behavior arguably but I didn’t get any error.

Also in terms of further development, yes a repo will be very helpful
if anyone else is interested in contributing. I used GitHub before
but I’m not sure if it feels comfortable for everyone — using it only
as a file hosting service makes me feel better but it does run whatever
software MS wants to run under the hood…


> On Oct 8, 2020, at 10:42 PM, Yuan Fu <casouri@gmail.com> wrote:
> 
> 
> 
>> On Oct 8, 2020, at 10:12 PM, Qiantan Hong <qhong@mit.edu> wrote:
>> 
>>> Another question: According to the algorithm, what happens when both cursor are at the same place and one inserts text? Is the other cursor pushed forward or not?
>> Currently the other cursor will not be pushed forward. Any suggestion for “better” behavior?
>> Also note that the cursor-treating part of version on this mailing list is not very polished.
>> I’ve made some fixes and improvements on my local version, will post shortly.
>> The new version will also contain Unicode support and active user list.
> 
> Pushing forward is more intuitive IMO. I asked because I see some small problems with the cursor overlay but I’m sure you’ve fixed them now.
> 
>> 
>>> Very cool! If I try to run crdt-test-client when already connected, Emacs creates a new buffer and compains "Error in post-command-hook (crdt--post-command): (wrong-type-argument stringp nil)”. Any idea why?
>> Thanks for the report! I haven’t consider this case. What do you think is a reasonable
>> behavior? Should it make a new connection?
> 
> It probably doesn’t make much sense to have multiple connections on a single Emacs instance to the same document. I’d just stop and complain to the user.
> 
> Yuan


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 360 bytes --]


> Pushing forward is more intuitive IMO. I asked because I see some small problems with the cursor overlay but I’m sure you’ve fixed them now.
I see. It’s possible to switch to that (in fact, relatively simple, just use the CRDT ID *after* the cursor
to label the position instead of before the cursor). I’ll probably do that in the next iteration.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - Wikipedia reference

[joakim]

Qiantan Hong <qhong@mit.edu> writes:

> Here is the current version, I feel like a relatively complete 
> set of essential features is implemented now.
> Changes:
> - Unicode support
> - fixed cursor behavior
> - M-x crdt-list-users displays a list of active users. Press enter
> to go to the position of that user’s cursor

Thanks for this work, I will try it within our team during one of our
online sessions.

I tried it briefly myself between 2 emacsen and it seemed to work
nicely.

/Joakim

>
>
>
> It seems that locally I didn’t reproduce yanking issue and multiple 
> crdt-test-client issue — it just open another same document normally,
> which is not the best behavior arguably but I didn’t get any error.
>
> Also in terms of further development, yes a repo will be very helpful
> if anyone else is interested in contributing. I used GitHub before
> but I’m not sure if it feels comfortable for everyone — using it only
> as a file hosting service makes me feel better but it does run whatever
> software MS wants to run under the hood…
>
>
>> On Oct 8, 2020, at 10:42 PM, Yuan Fu <casouri@gmail.com> wrote:
>> 
>> 
>> 
>>> On Oct 8, 2020, at 10:12 PM, Qiantan Hong <qhong@mit.edu> wrote:
>>> 
>>>> Another question: According to the algorithm, what happens when both cursor are at the same place and one inserts text? Is the other cursor pushed forward or not?
>>> Currently the other cursor will not be pushed forward. Any suggestion for “better” behavior?
>>> Also note that the cursor-treating part of version on this mailing list is not very polished.
>>> I’ve made some fixes and improvements on my local version, will post shortly.
>>> The new version will also contain Unicode support and active user list.
>> 
>> Pushing forward is more intuitive IMO. I asked because I see some small problems with the cursor overlay but I’m sure you’ve fixed them now.
>> 
>>> 
>>>> Very cool! If I try to run crdt-test-client when already connected, Emacs creates a new buffer and compains "Error in post-command-hook (crdt--post-command): (wrong-type-argument stringp nil)”. Any idea why?
>>> Thanks for the report! I haven’t consider this case. What do you think is a reasonable
>>> behavior? Should it make a new connection?
>> 
>> It probably doesn’t make much sense to have multiple connections on a single Emacs instance to the same document. I’d just stop and complain to the user.
>> 
>> Yuan
>
-- 
Joakim Verona
joakim@verona.se

Re: Question collaborative editing - Wikipedia reference

[Joe Corneli]

[-- Attachment #1: Type: text/plain, Size: 1139 bytes --]

Hi Qiantan and all,

As a basic solution for pairing — before your nice software appeared
recently — we were using lockstep.el.

One cool feature of lockstep is that you share the whole Emacs session.
This is useful for workflows that involve
lots of files and buffers (e.g., consider using it alongside Projectile, or
Org Roam).  Access to the shell or Helm is
transparent, for example.

One downside I noticed recently is that having multiple emacsclients
working on the same set of files seems to
interfere with Org Roam, with regard to syncing the database (... still
investigating that). This is a problem that
would not come up with crdt.el b/c it manages syncing at the buffer level,
not the file level!

All this leads to some possible feature requests for crdt.

(1) Would it make sense for a future version of crdt.el to also manage a
list of files (like Projectile or Gobby?).
(2) Would it make sense for a future version of crdt.el to sync not just
buffers... but also commands and windows (like lockstep?)

I realise it's still early days, so I don't want to distract w/ these
questions.

Joe

[-- Attachment #2: Type: text/html, Size: 1384 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 1905 bytes --]

> 
> All this leads to some possible feature requests for crdt.
> 
> (1) Would it make sense for a future version of crdt.el to also manage a list of files (like Projectile or Gobby?).  
> (2) Would it make sense for a future version of crdt.el to sync not just buffers... but also commands and windows (like lockstep?)
> 
Yes, I think both make perfect sense. 
For 1, if it’s to be implemented separately I’ll have to think about the right model.
One end of the spectrum is to have a built-in file manager — the other end
is to just provide some command like M-x crdt-goto-this-buffer to allow taking
a CRDT session to the buffer with another file.

I personally thought more about 2, and I concluded that supporting arbitrary command
in general is almost impossible (given that we want to apply command in real time, and
eventual consistency). When one site invoke a command X that depends on a chunk of text, 
some other site might mutate the text at the same time, or invoking some command that
interfere with X — and without any knowledge of how the command works it’s impossible
to solve this general problem.

However, I do think it’s possible to make other Emacs packages realtime-collaboration-complaint
with relatively little effort if I provide the right infrastructure. Emacs applications are in general
built around buffer, and by just syncing buffer text, text properties, markers and overlays using CRDT,
those without using any variables will work automatically — with eventual consistency guarantee!
Then I might provide some distributed version of DEFVARs for buffer-local-varaibles —
def-crdt-counter etc, or some non-realtime def-server-var — with a few annotation a single-Emacs
package should be able to be adapted to collaborative use.

I’m very excited by this idea, I might call it “Emacs as a distributed (or collaborative) OS”.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Jean Louis]

* Qiantan Hong <qhong@mit.edu> [2020-10-09 10:10]:
> Here is the current version, I feel like a relatively complete 
> set of essential features is implemented now.
> Changes:
> - Unicode support
> - fixed cursor behavior
> - M-x crdt-list-users displays a list of active users. Press enter
> to go to the position of that user’s cursor

Thank you for improvements.

Try to see this video, there are 2 issues that happened, at about 2:00
you will see first line was not synchronized, but later nothing was
synchronized at all

https://gnu.support/images/2020/10/2020-10-09/crdt-bug.ogv


Jean

Re: Question collaborative editing - Wikipedia reference

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 806 bytes --]

I see, that’s a very helpful footage! I’ll try to debug
this over the weekend.

> On Oct 9, 2020, at 1:46 PM, Jean Louis <bugs@gnu.support> wrote:
> 
> * Qiantan Hong <qhong@mit.edu> [2020-10-09 10:10]:
>> Here is the current version, I feel like a relatively complete 
>> set of essential features is implemented now.
>> Changes:
>> - Unicode support
>> - fixed cursor behavior
>> - M-x crdt-list-users displays a list of active users. Press enter
>> to go to the position of that user’s cursor
> 
> Thank you for improvements.
> 
> Try to see this video, there are 2 issues that happened, at about 2:00
> you will see first line was not synchronized, but later nothing was
> synchronized at all
> 
> https://gnu.support/images/2020/10/2020-10-09/crdt-bug.ogv
> 
> 
> Jean


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Karl Fogel]

[-- Attachment #1: Type: text/plain, Size: 790 bytes --]

On 09 Oct 2020, Qiantan Hong wrote:
>Also in terms of further development, yes a repo will be very helpful
>if anyone else is interested in contributing. I used GitHub before
>but I’m not sure if it feels comfortable for everyone — using it only
>as a file hosting service makes me feel better but it does run whatever
>software MS wants to run under the hood…

If you'd like to host at https://code.librehq.com/, please do.  It's fully free software, so at least that shouldn't be a concern for anyone.

(We set it up recently; the process of registering an account seems to work fine, but if you run into any troubles let me know and I'll help right away.  I'm moving all my own stuff over there, see e.g. https://code.librehq.com/kfogel/urlete/.)

Best regards,
-Karl

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Qiantan Hong]

[-- Attachment #1.1: Type: text/plain, Size: 1139 bytes --]

Thanks Karl! The project is now hosted at https://code.librehq.com/qhong/crdt.el <https://code.librehq.com/qhong/crdt.el>
Be sure to get the latest version.

Thanks Jean and Joe for testing and reporting.
I’ve caught and fixed two bugs.

> On Oct 9, 2020, at 4:31 PM, Karl Fogel <kfogel@red-bean.com> wrote:
> 
> On 09 Oct 2020, Qiantan Hong wrote:
>> Also in terms of further development, yes a repo will be very helpful
>> if anyone else is interested in contributing. I used GitHub before
>> but I’m not sure if it feels comfortable for everyone — using it only
>> as a file hosting service makes me feel better but it does run whatever
>> software MS wants to run under the hood…
> 
> If you'd like to host at https://code.librehq.com/, please do.  It's fully free software, so at least that shouldn't be a concern for anyone.
> 
> (We set it up recently; the process of registering an account seems to work fine, but if you run into any troubles let me know and I'll help right away.  I'm moving all my own stuff over there, see e.g. https://code.librehq.com/kfogel/urlete/.)
> 
> Best regards,
> -Karl


[-- Attachment #1.2: Type: text/html, Size: 1969 bytes --]

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 2391 bytes --]

I’m now experimenting those in my local copy.
(2) I’m able to synchronize overlays between Emacsen now,
and as a result, e.g. org-cycle now get synced between Emacsen.
There’s a few choice to make though:
- what’s a reasonable way to decide what overlays to sync?
   I’m currently deciding it based on the command that create the overlay
  (only overlay created during executing a customizable set of commands 
  are tracked). Is there better way?
- similar question to text property.

Also about (1), what do you think a a good format? I have
two design in mind:
a) share a full directory, probably making use of projectile (however
projectile is not in Emacs itself but I’d wish crdt.el get included in
the future)
b)\x01 share a “workspace” with flat namespace. Server can decide
which buffer to add to it. (And if they're UNIX fan they’ll write a function
to add a directory). I like flat namespace, but to make it useful
it’d better has some tagging mechanism, and I feel like there’s
too many wheels to reinvent. Another problem is how to allow
clients creating new file (maybe don’t?)

> On Oct 9, 2020, at 9:33 AM, Joe Corneli <holtzermann17@gmail.com> wrote:
> 
> Hi Qiantan and all,
> 
> As a basic solution for pairing — before your nice software appeared recently — we were using lockstep.el.
> 
> One cool feature of lockstep is that you share the whole Emacs session.  This is useful for workflows that involve
> lots of files and buffers (e.g., consider using it alongside Projectile, or Org Roam).  Access to the shell or Helm is
> transparent, for example.
> 
> One downside I noticed recently is that having multiple emacsclients working on the same set of files seems to
> interfere with Org Roam, with regard to syncing the database (... still investigating that). This is a problem that
> would not come up with crdt.el b/c it manages syncing at the buffer level, not the file level!
> 
> All this leads to some possible feature requests for crdt.
> 
> (1) Would it make sense for a future version of crdt.el to also manage a list of files (like Projectile or Gobby?).  
> (2) Would it make sense for a future version of crdt.el to sync not just buffers... but also commands and windows (like lockstep?)
> 
> I realise it's still early days, so I don't want to distract w/ these questions.
> 
> Joe


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> CC: "joakim@verona.se" <joakim@verona.se>, Yuan Fu <casouri@gmail.com>,
>         Fermin
> 	<fmfs@posteo.net>, Jean Louis <bugs@gnu.support>,
>         Caio Henrique
> 	<caiohcs0@gmail.com>, Ergus <spacibba@aol.com>,
>         Noam Postavsky
> 	<npostavs@gmail.com>, Karl Fogel <kfogel@red-bean.com>,
>         Stefan Monnier
> 	<monnier@iro.umontreal.ca>,
>         Eli Zaretskii <eliz@gnu.org>, Emacs developers
> 	<emacs-devel@gnu.org>
> Date: Sun, 11 Oct 2020 17:49:44 +0000
> 
> - what’s a reasonable way to decide what overlays to sync?
>    I’m currently deciding it based on the command that create the overlay
>   (only overlay created during executing a customizable set of commands 
>   are tracked). Is there better way?
> - similar question to text property.

Don't insert-in-front-hooks and insert-behind-hooks provide the way to
know which overlays and text properties changed?

Re: Question collaborative editing - Wikipedia reference

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 1737 bytes --]

It does, but I don’t think it make sense to synchronize *every*
overlay/text property changes. For example, syntax highlight
can be done locally and doesn’t need to be synced over Internet.
I’m using hightlight-line and highlight-parenthesis, those overlay don’t
need to be synced as well. There has to be some customizable filtering
(I was filtering based on what command created the overlays).

BTW in my implementation we don’t need those hooks to track
marker advancing movements, because CRDT-IDs are attached
to buffer text. For a advancing marker I just need to locate it
with the CRDT-ID at the right of it and it’ll always stay in the correct
location.

> On Oct 11, 2020, at 2:21 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Qiantan Hong <qhong@mit.edu>
>> CC: "joakim@verona.se" <joakim@verona.se>, Yuan Fu <casouri@gmail.com>,
>>        Fermin
>> 	<fmfs@posteo.net>, Jean Louis <bugs@gnu.support>,
>>        Caio Henrique
>> 	<caiohcs0@gmail.com>, Ergus <spacibba@aol.com>,
>>        Noam Postavsky
>> 	<npostavs@gmail.com>, Karl Fogel <kfogel@red-bean.com>,
>>        Stefan Monnier
>> 	<monnier@iro.umontreal.ca>,
>>        Eli Zaretskii <eliz@gnu.org>, Emacs developers
>> 	<emacs-devel@gnu.org>
>> Date: Sun, 11 Oct 2020 17:49:44 +0000
>> 
>> - what’s a reasonable way to decide what overlays to sync?
>>   I’m currently deciding it based on the command that create the overlay
>>  (only overlay created during executing a customizable set of commands 
>>  are tracked). Is there better way?
>> - similar question to text property.
> 
> Don't insert-in-front-hooks and insert-behind-hooks provide the way to
> know which overlays and text properties changed?


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Joe Corneli via Emacs development discussions.]

Qiantan Hong writes:

> I’m now experimenting those in my local copy.
> (2) I’m able to synchronize overlays between Emacsen now,
> and as a result, e.g. org-cycle now get synced between Emacsen.
> There’s a few choice to make though:
> - what’s a reasonable way to decide what overlays to sync?

I think it would be nice to always have the option, when using Org mode,
to have *different* folding properties on different clients.  E.g., if
my friend is editing inside of a node that I have folded, that’s no
problem.  To me that seems like a good fit for the multiple-cursor
editing paradigm.  But I can also see how syncing the folding could be
nice — probably along with a function to "summon" other cursors to where
I am, so that everyone can join up.  (This would also be nice in
connection with the other feature, in which people might be working
across multiple files.)

Maybe what could be introduced would be crdt-sync-overlays-mode, which
would be a minor mode that syncs all overlays when anyone turns it on?
Furthermore, whoever turns it on could propagate their overlays around?

(It would be interesting to look at use cases beyond Org mode: I have
one specific one one in mind... I’ll see if I can dust off the code, and
if it looks worth pursuing I’ll contact you about it off list!)

> b) share a “workspace” with flat namespace.

That seems like a good strategy: this would remove the direct dependency
on projectile, but there could be a simple compatibility package to
allow projectile to interact with the workspace.

> Another problem is how to allow clients creating new file (maybe
> don’t?)

This reminds me a little bit of org-roam-find-file.  In that case, files
are being synced to a local database, but in your case they’d be synced
around to a distributed database.  

>> (2) Would it make sense for a future version of crdt.el to sync not
>> just buffers... but also commands and windows (like lockstep?)

I was wondering, how much does it break things if you were to sync the
minibuffer?  Would that be a cheap way to sync commands, or a disaster
in the making...?

-- 
Dr Joseph A. Corneli (https://github.com/holtzermann17)

HYPERREAL ENTERPRISES LTD is a private company limited by shares, incorporated
25th, June 2019 as Company Number 634284 on the Register of Companies for
Scotland (https://beta.companieshouse.gov.uk/company/SC634284).

Re: Question collaborative editing - Wikipedia reference

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> CC: Joe Corneli <holtzermann17@gmail.com>,
>         "joakim@verona.se"
> 	<joakim@verona.se>, Yuan Fu <casouri@gmail.com>,
>         Fermin <fmfs@posteo.net>, Jean Louis <bugs@gnu.support>,
>         Caio Henrique <caiohcs0@gmail.com>, Ergus
> 	<spacibba@aol.com>,
>         Noam Postavsky <npostavs@gmail.com>, Karl Fogel
> 	<kfogel@red-bean.com>,
>         Stefan Monnier <monnier@iro.umontreal.ca>,
>         "Emacs
>  developers" <emacs-devel@gnu.org>
> Date: Sun, 11 Oct 2020 18:26:25 +0000
> 
> It does, but I don’t think it make sense to synchronize *every*
> overlay/text property changes. For example, syntax highlight
> can be done locally and doesn’t need to be synced over Internet.
> I’m using hightlight-line and highlight-parenthesis, those overlay don’t
> need to be synced as well. There has to be some customizable filtering
> (I was filtering based on what command created the overlays).

These hooks can do the filtering, no?

Re: Question collaborative editing - Wikipedia reference

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 2026 bytes --]

(Part of) it can be done there if we’ve decided on “how to filter”.
In the current way doing so is not required. But the question is
“how to filter”.

Let the user providing an arbitrary predicate (overlay->bool) seems to be
the most canonical, but it’s not. How to compose multiple predicates
provided by user? Or we can just let user do it and accept only one predicate.
But even so, apparently one piece of information missing is where the
overlay comes from. Lots of command don’t assign any ‘category
to the overlay their created and it’s hard to tell what the overlay is for
just by looking at the overlay object. In fact, currently I don’t use
any user provided predicate and filter solely based on what command
creates the overlay (by advising it to add a ‘crdt-meta property) and it
works well for org-mode folding, but I’m not sure if that works in other
scenarios.


> On Oct 11, 2020, at 2:56 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Qiantan Hong <qhong@mit.edu>
>> CC: Joe Corneli <holtzermann17@gmail.com>,
>>        "joakim@verona.se"
>> 	<joakim@verona.se>, Yuan Fu <casouri@gmail.com>,
>>        Fermin <fmfs@posteo.net>, Jean Louis <bugs@gnu.support>,
>>        Caio Henrique <caiohcs0@gmail.com>, Ergus
>> 	<spacibba@aol.com>,
>>        Noam Postavsky <npostavs@gmail.com>, Karl Fogel
>> 	<kfogel@red-bean.com>,
>>        Stefan Monnier <monnier@iro.umontreal.ca>,
>>        "Emacs
>> developers" <emacs-devel@gnu.org>
>> Date: Sun, 11 Oct 2020 18:26:25 +0000
>> 
>> It does, but I don’t think it make sense to synchronize *every*
>> overlay/text property changes. For example, syntax highlight
>> can be done locally and doesn’t need to be synced over Internet.
>> I’m using hightlight-line and highlight-parenthesis, those overlay don’t
>> need to be synced as well. There has to be some customizable filtering
>> (I was filtering based on what command created the overlays).
> 
> These hooks can do the filtering, no?


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> CC: Joe Corneli <holtzermann17@gmail.com>,
>         "joakim@verona.se"
> 	<joakim@verona.se>, Yuan Fu <casouri@gmail.com>,
>         Fermin <fmfs@posteo.net>, Jean Louis <bugs@gnu.support>,
>         Caio Henrique <caiohcs0@gmail.com>, Ergus
> 	<spacibba@aol.com>,
>         Noam Postavsky <npostavs@gmail.com>, Karl Fogel
> 	<kfogel@red-bean.com>,
>         Stefan Monnier <monnier@iro.umontreal.ca>,
>         emacs-devel
> 	<emacs-devel@gnu.org>
> Date: Sun, 11 Oct 2020 19:57:43 +0000
> 
> (Part of) it can be done there if we’ve decided on “how to filter”.
> In the current way doing so is not required. But the question is
> “how to filter”.
> 
> Let the user providing an arbitrary predicate (overlay->bool) seems to be
> the most canonical, but it’s not. How to compose multiple predicates
> provided by user? Or we can just let user do it and accept only one predicate.
> But even so, apparently one piece of information missing is where the
> overlay comes from. Lots of command don’t assign any ‘category
> to the overlay their created and it’s hard to tell what the overlay is for
> just by looking at the overlay object. In fact, currently I don’t use
> any user provided predicate and filter solely based on what command
> creates the overlay (by advising it to add a ‘crdt-meta property) and it
> works well for org-mode folding, but I’m not sure if that works in other
> scenarios.

Let me turn the table and ask why did you think there's a need to send
information about overlays and text properties over the wire?  What
were the use cases where you thought this would be necessary?

Re: Question collaborative editing - Wikipedia reference

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 2080 bytes --]

E.g. when two users are editing the same org document,
in some cases they may want two replica to fold the same sections.
— I always find myself wanting this feature to make sure both
are focusing on the same parts of the document, and at least seeing
the same thing.

> On Oct 11, 2020, at 10:26 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Qiantan Hong <qhong@mit.edu>
>> CC: Joe Corneli <holtzermann17@gmail.com>,
>>        "joakim@verona.se"
>> 	<joakim@verona.se>, Yuan Fu <casouri@gmail.com>,
>>        Fermin <fmfs@posteo.net>, Jean Louis <bugs@gnu.support>,
>>        Caio Henrique <caiohcs0@gmail.com>, Ergus
>> 	<spacibba@aol.com>,
>>        Noam Postavsky <npostavs@gmail.com>, Karl Fogel
>> 	<kfogel@red-bean.com>,
>>        Stefan Monnier <monnier@iro.umontreal.ca>,
>>        emacs-devel
>> 	<emacs-devel@gnu.org>
>> Date: Sun, 11 Oct 2020 19:57:43 +0000
>> 
>> (Part of) it can be done there if we’ve decided on “how to filter”.
>> In the current way doing so is not required. But the question is
>> “how to filter”.
>> 
>> Let the user providing an arbitrary predicate (overlay->bool) seems to be
>> the most canonical, but it’s not. How to compose multiple predicates
>> provided by user? Or we can just let user do it and accept only one predicate.
>> But even so, apparently one piece of information missing is where the
>> overlay comes from. Lots of command don’t assign any ‘category
>> to the overlay their created and it’s hard to tell what the overlay is for
>> just by looking at the overlay object. In fact, currently I don’t use
>> any user provided predicate and filter solely based on what command
>> creates the overlay (by advising it to add a ‘crdt-meta property) and it
>> works well for org-mode folding, but I’m not sure if that works in other
>> scenarios.
> 
> Let me turn the table and ask why did you think there's a need to send
> information about overlays and text properties over the wire?  What
> were the use cases where you thought this would be necessary?


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> CC: Joe Corneli <holtzermann17@gmail.com>,
>         "joakim@verona.se"
> 	<joakim@verona.se>, Yuan Fu <casouri@gmail.com>,
>         Fermin <fmfs@posteo.net>, Jean Louis <bugs@gnu.support>,
>         Caio Henrique <caiohcs0@gmail.com>, Ergus
> 	<spacibba@aol.com>,
>         Noam Postavsky <npostavs@gmail.com>, Karl Fogel
> 	<kfogel@red-bean.com>,
>         Stefan Monnier <monnier@iro.umontreal.ca>,
>         emacs-devel
> 	<emacs-devel@gnu.org>
> Date: Mon, 12 Oct 2020 03:22:55 +0000
> 
> E.g. when two users are editing the same org document,
> in some cases they may want two replica to fold the same sections.
> — I always find myself wanting this feature to make sure both
> are focusing on the same parts of the document, and at least seeing
> the same thing.

This seems to indicate you want the capability of sending commands,
not their results.  The command will then be executed on the remote,
and will produce the desired effects.

Re: Question collaborative editing - Wikipedia reference

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 2144 bytes --]

The problem with this is it will require considering conflict
resolution for every combination of command (which is basically
impossible) and it’s hard to even just guarantee eventual consistency.

One simple example to illustrate this:
* Level 1
** Level 2
some text

User 1 pressed TAB (to invoke org-cycle) on ** Level 2,
user 2 delete ** Level 2 in the same time.
If we just send (command,location) over the wire,
in the end user 1 will end up having a folded “some text”
without “**level 2” heading, and user1 will have level 1 folded.

With a overlay-synchronization approach, I was able to
support org-cycle with eventual consistency guarantee in 1 line
(just add to the filter). 
(The synchronization implementation itself is about 100 lines).
I was thinking that my way of filtering might not be general enough
but now I think there’s probably very few use cases for this —
org-cycle is a useful one — so I don’t need to worry much about
filtering customization. What do others think?

> On Oct 12, 2020, at 10:40 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Qiantan Hong <qhong@mit.edu>
>> CC: Joe Corneli <holtzermann17@gmail.com>,
>>        "joakim@verona.se"
>> 	<joakim@verona.se>, Yuan Fu <casouri@gmail.com>,
>>        Fermin <fmfs@posteo.net>, Jean Louis <bugs@gnu.support>,
>>        Caio Henrique <caiohcs0@gmail.com>, Ergus
>> 	<spacibba@aol.com>,
>>        Noam Postavsky <npostavs@gmail.com>, Karl Fogel
>> 	<kfogel@red-bean.com>,
>>        Stefan Monnier <monnier@iro.umontreal.ca>,
>>        emacs-devel
>> 	<emacs-devel@gnu.org>
>> Date: Mon, 12 Oct 2020 03:22:55 +0000
>> 
>> E.g. when two users are editing the same org document,
>> in some cases they may want two replica to fold the same sections.
>> — I always find myself wanting this feature to make sure both
>> are focusing on the same parts of the document, and at least seeing
>> the same thing.
> 
> This seems to indicate you want the capability of sending commands,
> not their results.  The command will then be executed on the remote,
> and will produce the desired effects.


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Yuan Fu]

> On Oct 12, 2020, at 12:53 PM, Qiantan Hong <qhong@mit.edu> wrote:
> 
> The problem with this is it will require considering conflict
> resolution for every combination of command (which is basically
> impossible) and it’s hard to even just guarantee eventual consistency.
> 
> One simple example to illustrate this:
> * Level 1
> ** Level 2
> some text
> 
> User 1 pressed TAB (to invoke org-cycle) on ** Level 2,
> user 2 delete ** Level 2 in the same time.
> If we just send (command,location) over the wire,
> in the end user 1 will end up having a folded “some text”
> without “**level 2” heading, and user1 will have level 1 folded.
> 
> With a overlay-synchronization approach, I was able to
> support org-cycle with eventual consistency guarantee in 1 line
> (just add to the filter). 
> (The synchronization implementation itself is about 100 lines).
> I was thinking that my way of filtering might not be general enough
> but now I think there’s probably very few use cases for this —
> org-cycle is a useful one — so I don’t need to worry much about
> filtering customization. What do others think?

Maybe define a profile system? Server can specify what to sync: org overlay, read-only properties, etc. And implement the synchronization as a plug-in. Note that some overlays are mananged by packages (i.e., keeps a pointer to the overlay) so syncing other overlays could leads to problems.

Yuan

Re: Question collaborative editing - Wikipedia reference

[Ergus]

On Tue, Oct 06, 2020 at 08:32:28PM +0300, Jean Louis wrote:
>* Qiantan Hong <qhong@mit.edu> [2020-10-06 04:04]:
>> > But this has 3 main problems.
>> >
>> > 1) On one hand such services require some servers (to work like google
>> > spreadsheet) and need to be provided somehow... something difficult as I
>> > don't think gnu or fsf have resources to maintain a service like that
>> > and provide it.
>
>Comment on crdt.el in the regard to above quote is that the server is
>on Emacs, so there need not be and should not be any third party to
>provide any service to provide service to Emacs, as Emacs is providing
>server through crdt.el
>
>Please look at how Gobby real time simultaneous editor is working,
>that is very similar now to crdt.el
>
>If user have Emacs with public IP address, all such user need to do is
>to tell the IP address and port and password to the collaborator, they
>can already connect.
>
>If user do not have public IP address but user is behind the router,
>then such user who connects over network or administrator who is
>helping users to connect over the network is assumed to have
>sufficient knowledge how to click few times or setup router to port
>forward from local network to public network.
>
>If user has VPS or dedicated server, then such can use ssh to port
>forward to the public server, or can use permanent VPN to connect to
>other collaborators which is anyway best setup. When I say permanent
>VPN, I mean "private" network and not VPN through third parties.
>
>To use VPS, today it is about $5 or less, expenses are not high.
>
>And let us not forget that there are many organizations where such
>collaborative simultaneous editing can take place who need not have
>any Internet connection, they may have their hotspots for local
>network or ethernet wire networks.
>
Not all the users have the skills/knowledge to do this. Even if they do,
the most frequent is that they don't have the privileges for that in
their network/work/home/ISP to open a port or so. Users with real IP are
not common these days.

VPS is an option... but do you really think that most users can/want to
create a virtual machine create a VPS + VPN and so just to edit a
document?... actually in some countries (like mine) the common VPS
services are blocked too...

>> > 2) On the other hand it will be better if the service is somehow
>> > distributed in order to give more privacy-security but also to reduce
>> > the load of the servers...
>
>We speak here of editing of a file, not transfer of huge binaries with
>many users. So there is no expected high load on the server. I cannot
>possibly understand why should simultaneous real time editing be
>distributed, maybe you think in similar fashion like Tor is doing it,
>but that is all definitely not necessary for Emacs to handle. You can
>handle your routing anyway through Tor, I am sure it will work, and
>you can route traffic through VPN, SSH and maybe other means.
>
Not Tor. Tor is an onion infrastructure I was referring to the user
connections. I explicitly said p2p so, direct communication without
needing an permanent intermediate server.

Many ISP don't allow you to connect directly or open ports to your
home. Some networks have fireworks. Some of them even change your IP
every some hours that's why services like noip need a program to ping
frequently with updated information.

>> > I still can't find any infrastructure we can
>> > use, cause most of the peer-to-peer libraries are for C++, javascript,
>> > Node.js and so on (example: webrtc). Just on yesterday I found
>> > n2n... But I am not a web specialist so it requires a lot of
>> > experimenting time for me.
>
>For crdt.el is definitely not needed, just as in Gobby editor is not
>needed. Collaborators who connect over Internet will have enough
>networking skills to connect to each other by using methods as I have
>explained above. Simplest method is ssh, and if any of collaborators
>have public IP exposed, such could be preferred server, and everybody
>else could connect to such, by doing simple ssh command with port
>forwarding, and editing afterwards.
>

I understand that crdt.el does a good work for local networks, but to
extend it ever internet your solutions are very limited. There is
something else needed and what you explained is exactly the kind of
things I won't recommend/suggest to any user to do (create a virtual
machine, install a vpn, create it, open a port in your router, get an
ISP with fix IP...). It is almost like open a tmux session over
ssh... will be simpler.

Specially with the inexpert users in mind... they can edit easily with
many other tools these days (either documents or code). So the idea is
to offer something "competitive"... not to challenge their skills and
make collaborative editing like a final lever of a complex videogame...

>> > 3) The other workflow (create a local server for others) is the
>> > "simplest" approach at the moment. But that is a problem for many use
>> > cases due to dynamic ip addreses, firewalls, opening ports and so on. It
>> > is fine for a class room or company, but not for working from  home.
>
>All solutions for that already exists, see above.
>
What you mention are not solutions... just workarounds and their
complexity (and requirements of skills) just show that some extra work
is needed to easy the user experience...

>> On this topic, I'm considering supporting sending the traffic over
>> IRC. Seems that it solves all those problem, what do you guys think?
>
>I think I feel like "abused" when I hear that.
>
>> The process will be that one user create a channel with a random
>> name, say on freenode.net, then they share the channel name
>> with other user (maybe via IRC as well!). Others can then join the
>> channel, and it behaves basically like TCP. To avoid spamming
>> the same authentication protocol for TCP (to be implemented) can
>> also work on IRC. The messages from user without authentication
>> are simply discarded.
>
>Please no.

Why not?


>:-)
>
>
>Jean
>
Ergus

Re: Question collaborative editing - Wikipedia reference

[Aurélien Aptel]

On Sun, Oct 4, 2020 at 7:56 PM Jean Louis <bugs@gnu.support> wrote:
> As it was in the Mother of All Demo by Douglas Engelbart, it is pity
> that Emacs yet does not support such, especially because Emacs can
> support such through its Lisp.

I don't think it's free software and I've never tried it but there is
this cross-editor plugin Floobits that allows multiple people to edit
interactively the same document from different editors (including
Emacs):

https://floobits.com/

Re: Question collaborative editing - Wikipedia reference

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 148 bytes --]

What about stuff like ngrok?
I see some reddit ppl mention it.
Just a single command
$ ngrok tcp 6530
Then it will give you a public accessible URL.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Jean Louis]

* Ergus <spacibba@aol.com> [2020-10-20 19:16]:
> > If user have Emacs with public IP address, all such user need to do is
> > to tell the IP address and port and password to the collaborator, they
> > can already connect.
> > 
> > If user do not have public IP address but user is behind the router,
> > then such user who connects over network or administrator who is
> > helping users to connect over the network is assumed to have
> > sufficient knowledge how to click few times or setup router to port
> > forward from local network to public network.
> > 
> > If user has VPS or dedicated server, then such can use ssh to port
> > forward to the public server, or can use permanent VPN to connect to
> > other collaborators which is anyway best setup. When I say permanent
> > VPN, I mean "private" network and not VPN through third parties.
> > 
> > To use VPS, today it is about $5 or less, expenses are not high.
> > 
> > And let us not forget that there are many organizations where such
> > collaborative simultaneous editing can take place who need not have
> > any Internet connection, they may have their hotspots for local
> > network or ethernet wire networks.

> Not all the users have the skills/knowledge to do this. Even if they do,
> the most frequent is that they don't have the privileges for that in
> their network/work/home/ISP to open a port or so. Users with real IP are
> not common these days.

I am comparing it with Gobby collaborative editor, it works, but it
does not say how to connect. Person who invites others to connect, I
am assuming, will learn how to do it.

With Qiantan we connected together and I was using ssh connection to
VPS.

> VPS is an option... but do you really think that most users can/want to
> create a virtual machine create a VPS + VPN and so just to edit a
> document?... actually in some countries (like mine) the common VPS
> services are blocked too...

You mean VPN services are blocked. As VPS is just virtual machine running.

I think that collaborative editing is first very usable in local area
networks, then over Internet. We have to keep in mind that Emacs is
installed on multi user computers. The daemon and client model are
already well done and I somehow think that multiple users connecting
through network to same computer could use emacsclient model to
collaborate. It is just not handy or very secure.

Now we have crdt.el and it can be used well in local area network. If
that is so, users already have IP address.

If it is not so, at least one of them has to have IP address exposed
to Internet that can expose also the server ports.

In gaming world this is very common, even children learn today how to
expose or forward ports from inside to the router, and they host
gaming servers.

> Not Tor. Tor is an onion infrastructure I was referring to the user
> connections. I explicitly said p2p so, direct communication without
> needing an permanent intermediate server.
> 
> Many ISP don't allow you to connect directly or open ports to your
> home. Some networks have fireworks. Some of them even change your IP
> every some hours that's why services like noip need a program to ping
> frequently with updated information.

That is all true. Though it will not prevent those who wish to collaborate.

If you have particular need, you can tell me, and I may tell you how
to connect. 

> I understand that crdt.el does a good work for local networks, but to
> extend it ever internet your solutions are very limited.

ssh today is very common, all VPS providers teach people how to
connect with SSH and it is not hard. I have enabled TCP forwarding and
GatewayPorts on server, and I could -R
LOCALPORT:my.example.com:REMOTE-PORT my.example.com already forward
the ports to Qiantan, and he could connect.

How people connect is quite different subject, that must be left to
people to decide, crdt.el cannot do much about it.

There are free shell accounts too.

> There is
> something else needed and what you explained is exactly the kind of
> things I won't recommend/suggest to any user to do (create a virtual
> machine, install a vpn, create it, open a port in your router, get an
> ISP with fix IP...). It is almost like open a tmux session over
> ssh... will be simpler.

I know my friend in Tacoma, WA, they connect over Wi-Fi to Internet,
then they collaborate on Google, that is why I favour collaboration in
Emacs, that I can offer him fully free software as solution.

You can be connected by local area network, or through wireless local
area network, through VPN, through SSH, by using public IP of one of
the participants, anything is possible.

My suggestions are examples. If you have particular need, you may tell
me, I can analyse it and give you few solutions how to connect. 

> Specially with the inexpert users in mind... they can edit easily with
> many other tools these days (either documents or code). So the idea is
> to offer something "competitive"... not to challenge their skills and
> make collaborative editing like a final lever of a complex videogame...

What Gobby editor did for years, now is possible on Emacs. We have
simultaneous real-time collaborative editing feature as free software.
If it is competitive or not, why? We do not have yet users' requests
to know what it has to be competitive. Qiantan is doing good job and
have enabled all general options, soon it will be finished.

Give me some examples, to what exactly would crdt.el need to be competitive?

What I can think of is that somebody can provide service of Emacs
collaboration, that would not be hard to do. I would make simple HTML
interface, users could launch their session and server would start on
specific port and by specific session name, and other users could
connect.

It is similar like Jitsi, everybody can connect on video call with
Jitsi, and somebody is providing a server. If people want they can
host it themselves.

> > > > 3) The other workflow (create a local server for others) is the
> > > > "simplest" approach at the moment. But that is a problem for many use
> > > > cases due to dynamic ip addreses, firewalls, opening ports and so on. It
> > > > is fine for a class room or company, but not for working from  home.
> > 
> > All solutions for that already exists, see above.
> > 
> What you mention are not solutions... just workarounds and their
> complexity (and requirements of skills) just show that some extra work
> is needed to easy the user experience...

OK you tell me your particular case problem and I will tell you how to connect.

Re: Question collaborative editing - Wikipedia reference

[Jean Louis]

* Aur??lien Aptel <aurelien.aptel+emacs@gmail.com> [2020-10-20 19:37]:
> On Sun, Oct 4, 2020 at 7:56 PM Jean Louis <bugs@gnu.support> wrote:
> > As it was in the Mother of All Demo by Douglas Engelbart, it is pity
> > that Emacs yet does not support such, especially because Emacs can
> > support such through its Lisp.
> 
> I don't think it's free software and I've never tried it but there is
> this cross-editor plugin Floobits that allows multiple people to edit
> interactively the same document from different editors (including
> Emacs):
> 
> https://floobits.com/

I would never use it, for reason that it is centralized server and I
do not know who are those people, why they do it, probably for money
and not to help people to control their computing, they compute for
people. But I would never use it primarily for reason that I cannot
know what are they doing with my data. It is unthinkable.

After looking into Emacs package for floobits, I can see it is made by
company Floobits, and it uses Python, like Emacs Lisp cannot connect
to their API, I don't believe.

Commands like M-x floobits-share-dir-private <RET> DIR <RET> are
supposed to to go over their server?! Ah no.

crdt.el from Qiantan is empowering users, they need no floobits
servers and whatever insecurities involved with it.

Re: Question collaborative editing - Wikipedia reference

[Jean Louis]

* Qiantan Hong <qhong@mit.edu> [2020-10-20 20:12]:
> What about stuff like ngrok?
> I see some reddit ppl mention it.
> Just a single command
> $ ngrok tcp 6530
> Then it will give you a public accessible URL.

That is proprietary software using centralized server. 

Why trust third party with data or use proprietary software?

Re: Question collaborative editing - Wikipedia reference

[Eli Zaretskii]

Any idea why we are getting email messages from you now that you wrote
6 days ago?

too many

[Jean Louis]

* Eli Zaretskii <eliz@gnu.org> [2020-10-26 22:47]:
> Any idea why we are getting email messages from you now that you wrote
> 6 days ago?

Sorry for that. Some emails were written in the offline queues on
multiple computers and sent by first Internet connection to outside
world. I will put attention on it.

Re: Question collaborative editing - Wikipedia reference

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 728 bytes --]

ngrok is free software I think, both server and client
side. However last time I check it myself it does
seems that you need to sign up to use TCP tunnel
on their official servers and the sign up page require
non-free JS. That’s why I didn’t mention it in my documentation.
> On Oct 20, 2020, at 2:51 PM, Jean Louis <bugs@gnu.support> wrote:
> 
> * Qiantan Hong <qhong@mit.edu> [2020-10-20 20:12]:
>> What about stuff like ngrok?
>> I see some reddit ppl mention it.
>> Just a single command
>> $ ngrok tcp 6530
>> Then it will give you a public accessible URL.
> 
> That is proprietary software using centralized server. 
> 
> Why trust third party with data or use proprietary software?
> 
> 
> 
> 


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing.

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 1402 bytes --]

Hi, now that I’ve almost done with other part I’m
looking back into this connectivity problem.

I mentioned the idea of messaging over IRC, however
after thinking a bit more on that approach, seems that
IRC has no reliability guarantee and I’ll have to implement
tcp-over-irc.el … that already seems like much more work
than expected.

ICE/STUN looks like the usual way to do the NAT traversal,
however I didn’t find any tcp tunnel over ICE/STUN that
I can use directly in Emacs. Does any one know any?

There’s libnice but it’s a C library, we’ll either have to
write more Emacs C code to use it or wrap it in a cli tool
providing tcp tunneling...

> On Sep 29, 2020, at 12:29 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>>> 1) On one hand such services require some servers (to work like google
>>> spreadsheet) and need to be provided somehow... something difficult as I
>>> don't think gnu or fsf have resources to maintain a service like that
>>> and provide it.
>> CRDT should do it.
> 
> I think CRDT make no difference to the problem referred to here, which
> is that two machines, each behind its own firewall will have difficulty
> talking to each other unless they go through some external server (at
> least to set up the initial connection).  There are *partial* solutions in
> the form of ICE/STUN.
> 
> 
>        Stefan
> 


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing.

[Jean Louis]

* Qiantan Hong <qhong@mit.edu> [2020-10-27 08:11]:
> Hi, now that I’ve almost done with other part I’m
> looking back into this connectivity problem.
> 
> I mentioned the idea of messaging over IRC, however
> after thinking a bit more on that approach, seems that
> IRC has no reliability guarantee and I’ll have to implement
> tcp-over-irc.el … that already seems like much more work
> than expected.
> 
> ICE/STUN looks like the usual way to do the NAT traversal,
> however I didn’t find any tcp tunnel over ICE/STUN that
> I can use directly in Emacs. Does any one know any?

Just as Gobby editor, it leaves connecting issues to the
administrator, it does not solves it itself. It is external issue.

Collaboration will be first taking place within local networks within
organizations. For example people connected to Wi-Fi access points or
wired networks.

Then between persons just as gaming peer to peer, it is possible to
expose port on the router.

Then there are solutins with tunneling and SSH port forwarding, then
directly by setting up Emacs as server remotely without port forward.

-- 
Jean Louis

Re: Question collaborative editing - Wikipedia reference

[Mathias Dahl]

[-- Attachment #1: Type: text/plain, Size: 340 bytes --]

Sorry for coming late to the party, and for this naïve and perhaps repeated
question/suggestion (I tried searching the archive but could not find any
efficient or reliable way), but can some service running on Savannah or
Gnu.org act as the central point, for those cases when the collaborators
cannot easily connect by other means?'

[-- Attachment #2: Type: text/html, Size: 361 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Qiantan Hong]

[-- Attachment #1.1: Type: text/plain, Size: 2611 bytes --]

> Just as Gobby editor, it leaves connecting issues to the
> administrator, it does not solves it itself. It is external issue.

We might be able to automate this as best effort and eliminate
the need for any network knowledge/expertise for most users.
I will explain below.

> Sorry for coming late to the party, and for this naïve and perhaps repeated question/suggestion (I tried searching the archive but could not find any efficient or reliable way), but can some service running on Savannah or Gnu.org <http://gnu.org/> act as the central point, for those cases when the collaborators cannot easily connect by other means?'

I don’t think that’s an ideal way to do it, unless we really don’t know
how to do it by other means, for the followingreasons:

- I assumes either Savannah or Gnu.org <http://gnu.org/> server is just a single server
with a fixed geolocation on the planet. It might work well for computers
around it, but the latency will be huge if the computer happen to be
in the other hemisphere from our server. CDN can’t help anything here.

- This will overlay all the traffic through GNU server, which is unnecessary
in lots of cases (e.g. when it’s possible to use STUN to traverse the NATs).
That will put maintenance burden on the server admin I think

I’m roaming through relevant material recently and I think I have those
options in mind

- use libnice to do the NAT traversal job. It will be convenient if there’s
any TCP tunneling tool built on libnice. Nobody mention any so far, so
maybe we need to implement one in C.

- use ipfs p2p tunneling functionality 
https://docs.ipfs.io/reference/cli/#ipfs-p2p <https://docs.ipfs.io/reference/cli/#ipfs-p2p>
I can immediately add some elisp to automate this such that user just issue
the new session Emacs command, then get a accessible IPFS URI,
if we think this is the way to go. I dislike the IPFS implementation however
because it’s return in a UNIXer language whose name shall not be mentioned.
Also IPFS is not a GNU project.

- use Tor hidden service.
It happens to traverse NAT (or some firewall) as a byproduct of its anonymity.
Some drawback
   + we also pay the anonymity tax - lower bandwidth and higher latency,
      because of the relays between.
   + starting hidden service require changing some config file and usually
      require root access, which make it a bit more cumbersome to automate
      in Elisp.

- GNUnet. I wish we were running our whole Internet over it. However 
   I doubt gnunet-cadet is usable with acceptable latency right now.



[-- Attachment #1.2: Type: text/html, Size: 3785 bytes --]

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Any idea why we are getting email messages from you now that you wrote
  > 6 days ago?

Look at the Received headers; maybe you will see which machine
they were stuck in.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Question collaborative editing - Wikipedia reference

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: bugs@gnu.support, emacs-devel@gnu.org
> Date: Wed, 28 Oct 2020 00:45:58 -0400
> 
>   > Any idea why we are getting email messages from you now that you wrote
>   > 6 days ago?
> 
> Look at the Received headers; maybe you will see which machine
> they were stuck in.

I did, before asking my question.  Guess what I've found out.

Re: Question collaborative editing - Wikipedia reference

[Jean Louis]

* Eli Zaretskii <eliz@gnu.org> [2020-10-28 18:09]:
> > From: Richard Stallman <rms@gnu.org>
> > Cc: bugs@gnu.support, emacs-devel@gnu.org
> > Date: Wed, 28 Oct 2020 00:45:58 -0400
> > 
> >   > Any idea why we are getting email messages from you now that you wrote
> >   > 6 days ago?
> > 
> > Look at the Received headers; maybe you will see which machine
> > they were stuck in.
> 
> I did, before asking my question.  Guess what I've found out.

I am also curious...

Re: Question collaborative editing - tools to connect

[Jean Louis]

* Qiantan Hong <qhong@mit.edu> [2020-10-26 23:37]:
> ngrok is free software I think, both server and client
> side. However last time I check it myself it does
> seems that you need to sign up to use TCP tunnel
> on their official servers and the sign up page require
> non-free JS. That’s why I didn’t mention it in my documentation.

SSH is easiest


3 stunnel -- should be accompanied with SSH
=========

  Beschreibung : A program that allows you to encrypt arbitrary TCP
  connections inside SSL, with libressl and OpenRC support URL :
  <https://www.stunnel.org/>


4 tuntox 
========

  Beschreibung : Tunnel TCP connections over the Tox protocol, with
  logger recommendation and OpenRC support URL :
  <https://github.com/gjedeer/tuntox>


1 iodine 
========

  Name : iodine Beschreibung : Tunnel IPv4 data through a DNS server URL
  : <http://code.kryo.se/iodine>

  It can be used through DNS, very handy when mobile data is out of
  credit but one has access to DNS system. Works with many
  providers. That would be collaborative avoidance of Internet fees.


2 ptunnel
=========

  Beschreibung : A tool for reliably tunneling TCP connections over ICMP
  echo request and reply packets URL :
  <http://www.cs.uit.no/~daniels/PingTunnel>

  Similar like above iodined.


-- 
Jean Louis

Re: Question collaborative editing - Wikipedia reference

[Eli Zaretskii]

> Date: Wed, 28 Oct 2020 19:20:15 +0300
> From: Jean Louis <bugs@gnu.support>
> Cc: rms@gnu.org, emacs-devel@gnu.org
> 
> * Eli Zaretskii <eliz@gnu.org> [2020-10-28 18:09]:
> > > From: Richard Stallman <rms@gnu.org>
> > > Cc: bugs@gnu.support, emacs-devel@gnu.org
> > > Date: Wed, 28 Oct 2020 00:45:58 -0400
> > > 
> > >   > Any idea why we are getting email messages from you now that you wrote
> > >   > 6 days ago?
> > > 
> > > Look at the Received headers; maybe you will see which machine
> > > they were stuck in.
> > 
> > I did, before asking my question.  Guess what I've found out.
> 
> I am also curious...

You mean, you cannot look into the headers of the messages you
yourself sent?

Anyway, I found the obvious: the messages were stuck in your local
machine, for some reason.  Once they got sent to the first email
server, they went all the way.

Re: Question collaborative editing - tools to connect

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 1733 bytes --]

We were talking about the case when both users
don’t have public accessible IP, neither VPS,
then ssh, iodine and ptunnel falls out.
Tuntox looks promising.
Any preference between IPFS, Tuntox and Tor hidden service?

> On Oct 28, 2020, at 12:29 PM, Jean Louis <bugs@gnu.support> wrote:
> 
> * Qiantan Hong <qhong@mit.edu> [2020-10-26 23:37]:
>> ngrok is free software I think, both server and client
>> side. However last time I check it myself it does
>> seems that you need to sign up to use TCP tunnel
>> on their official servers and the sign up page require
>> non-free JS. That’s why I didn’t mention it in my documentation.
> 
> SSH is easiest
> 
> 
> 3 stunnel -- should be accompanied with SSH
> =========
> 
>  Beschreibung : A program that allows you to encrypt arbitrary TCP
>  connections inside SSL, with libressl and OpenRC support URL :
>  <https://www.stunnel.org/>
> 
> 
> 4 tuntox 
> ========
> 
>  Beschreibung : Tunnel TCP connections over the Tox protocol, with
>  logger recommendation and OpenRC support URL :
>  <https://github.com/gjedeer/tuntox>
> 
> 
> 1 iodine 
> ========
> 
>  Name : iodine Beschreibung : Tunnel IPv4 data through a DNS server URL
>  : <http://code.kryo.se/iodine>
> 
>  It can be used through DNS, very handy when mobile data is out of
>  credit but one has access to DNS system. Works with many
>  providers. That would be collaborative avoidance of Internet fees.
> 
> 
> 2 ptunnel
> =========
> 
>  Beschreibung : A tool for reliably tunneling TCP connections over ICMP
>  echo request and reply packets URL :
>  <http://www.cs.uit.no/~daniels/PingTunnel>
> 
>  Similar like above iodined.
> 
> 
> -- 
> Jean Louis


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - Wikipedia reference

[Jean Louis]

* Eli Zaretskii <eliz@gnu.org> [2020-10-28 19:34]:
> > Date: Wed, 28 Oct 2020 19:20:15 +0300
> > From: Jean Louis <bugs@gnu.support>
> > Cc: rms@gnu.org, emacs-devel@gnu.org
> > 
> > * Eli Zaretskii <eliz@gnu.org> [2020-10-28 18:09]:
> > > > From: Richard Stallman <rms@gnu.org>
> > > > Cc: bugs@gnu.support, emacs-devel@gnu.org
> > > > Date: Wed, 28 Oct 2020 00:45:58 -0400
> > > > 
> > > >   > Any idea why we are getting email messages from you now that you wrote
> > > >   > 6 days ago?
> > > > 
> > > > Look at the Received headers; maybe you will see which machine
> > > > they were stuck in.
> > > 
> > > I did, before asking my question.  Guess what I've found out.
> > 
> > I am also curious...
> 
> You mean, you cannot look into the headers of the messages you
> yourself sent?
> 
> Anyway, I found the obvious: the messages were stuck in your local
> machine, for some reason.  Once they got sent to the first email
> server, they went all the way.

That computer is in other place and I did not transfer sent files to
my main computer.

Re: Question collaborative editing - crdt.el, does it use SSL?

[Jean Louis]

* Qiantan Hong <qhong@mit.edu> [2020-10-28 19:41]:
> We were talking about the case when both users
> don’t have public accessible IP, neither VPS,
> then ssh, iodine and ptunnel falls out.
> Tuntox looks promising.
> Any preference between IPFS, Tuntox and Tor hidden service?

All of those methods shall be tested and I can test it again with
you, after testing it is possible to say something what works best.

What we need is low latency speedy methods that do not break.

All of connection methods should be mentioned in manual.

Stunnel is for securing the connection unless you already use secure
connections, do you?

-- 
Jean Louis

Re: Question collaborative editing - torify emacs

[Jean Louis]

* Qiantan Hong <qhong@mit.edu> [2020-10-28 19:57]:
> We were talking about the case when both users
> don’t have public accessible IP, neither VPS,
> then ssh, iodine and ptunnel falls out.
> Tuntox looks promising.
> Any preference between IPFS, Tuntox and Tor hidden service?

I have tried:

$ torify emacs

and Duckduckgo says: Your IP address is unavailable, great. Now I have
safety and allegedly no IP address. On the other website I could
obtain it.

I am not sure if Tor and torify would be changing the IP address
during the connection, that is something to be verified, if it is so,
then it is poor choice for collaborative editing.

Emacs browsing over Tor works well, and on the other side I found the
IP address.

You can try to torify emacs, open up server, tell me the IP address
(other websites will tell it) and I can try to connect.

-- 
Jean Louis

Re: Question collaborative editing - torify emacs

[Qiantan Hong]

[-- Attachment #1.1: Type: text/plain, Size: 646 bytes --]

> 
> You can try to torify emacs, open up server, tell me the IP address
> (other websites will tell it) and I can try to connect.
That’s not how tor work. I’m pretty sure this IP will change,
and will not accept connection.

What’s supposed to happen is the server starts a Tor hidden service
https://roll.urown.net/server/tor/tor-hidden-service.html <https://roll.urown.net/server/tor/tor-hidden-service.html>
and the client connect to it through the Tor proxy installed on the
client machine.
There’s a lot to setup, but if user gives Emacs root access
it won’t be that hard to write an Elisp script to automate all these.

[-- Attachment #1.2: Type: text/html, Size: 1151 bytes --]

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - torify emacs

[Jean Louis]

* Qiantan Hong <qhong@mit.edu> [2020-10-28 21:04]:
> > 
> > You can try to torify emacs, open up server, tell me the IP address
> > (other websites will tell it) and I can try to connect.
> That’s not how tor work. I’m pretty sure this IP will change,
> and will not accept connection.
> 
> What’s supposed to happen is the server starts a Tor hidden service
> https://roll.urown.net/server/tor/tor-hidden-service.html <https://roll.urown.net/server/tor/tor-hidden-service.html>
> and the client connect to it through the Tor proxy installed on the
> client machine.
> There’s a lot to setup, but if user gives Emacs root access
> it won’t be that hard to write an Elisp script to automate all these.

It does not sound safe: Emacs with root access on network?

-- 
Jean Louis

Re: Question collaborative editing - crdt.el, does it use SSL?

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 795 bytes --]

> 
> All of those methods shall be tested and I can test it again with
> you, after testing it is possible to say something what works best.
> 
> What we need is low latency speedy methods that do not break.
> 
> All of connection methods should be mentioned in manual.
Also I think we can make it a lot easier for users if I put in
more Elisp code, and 
- have a list of those method for server to choose from when creating
new session
- let the server copy a URI-style link and let crdt-connect recognize it
(e.g. tuntox://… ipfs://… onion://…)

> Stunnel is for securing the connection unless you already use secure
> connections, do you?
I haven’t, because I didn’t find a way to let Emacs network-process
server use TLS.
Setting up stunnel seems to require root?

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - crdt.el, does it use SSL?

[Jean Louis]

* Qiantan Hong <qhong@mit.edu> [2020-10-29 00:52]:
> > Stunnel is for securing the connection unless you already use secure
> > connections, do you?
> I haven’t, because I didn’t find a way to let Emacs network-process
> server use TLS.
> Setting up stunnel seems to require root?

Stunnel is probably irrelevant for many as Emacs can connect over SSL.

Please look into eww as it uses SSL. Like open-network-stream and
gnutls- functions, please C-h f gnutls-negotiate and you will get more
references.

-- 
Jean Louis

Re: Question collaborative editing - crdt.el, does it use SSL?

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 768 bytes --]

I know that make-network-process can make TLS connection,
however it seems that it only works when Emacs is behaving as client.

> On Oct 28, 2020, at 6:36 PM, Jean Louis <bugs@gnu.support> wrote:
> 
> * Qiantan Hong <qhong@mit.edu> [2020-10-29 00:52]:
>>> Stunnel is for securing the connection unless you already use secure
>>> connections, do you?
>> I haven’t, because I didn’t find a way to let Emacs network-process
>> server use TLS.
>> Setting up stunnel seems to require root?
> 
> Stunnel is probably irrelevant for many as Emacs can connect over SSL.
> 
> Please look into eww as it uses SSL. Like open-network-stream and
> gnutls- functions, please C-h f gnutls-negotiate and you will get more
> references.
> 
> -- 
> Jean Louis


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - tools to connect

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > We were talking about the case when both users
  > don’t have public accessible IP, neither VPS,
  > then ssh, iodine and ptunnel falls out.
  > Tuntox looks promising.
  > Any preference between IPFS, Tuntox and Tor hidden service?

Can people find someone who is an expert on addressing these challenges
to peer-to-peer communication?  The expert might help us avoid
reinventing lots of wheels.


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Question collaborative editing - crdt.el, does it use SSL?

[Jean Louis]

* Qiantan Hong <qhong@mit.edu> [2020-10-29 02:09]:
> I know that make-network-process can make TLS connection,
> however it seems that it only works when Emacs is behaving as
> client.

You tried with these?

:plist PLIST -- Install PLIST as the new process’s initial plist.
:tls-parameters LIST -- is a list that should be supplied if you’re
opening a TLS connection.  The first element is the TLS type (either
‘gnutls-x509pki’ or ‘gnutls-anon’), and the remaining elements should
be a keyword list accepted by gnutls-boot (as returned by
‘gnutls-boot-parameters’).

:server QLEN -- if QLEN is non-nil, create a server process for the
specified FAMILY, SERVICE, and connection type (stream or datagram).
If QLEN is an integer, it is used as the max. length of the server’s
pending connection queue (also known as the backlog); the default
queue length is 5.  Default is to create a client process.

-- 
Jean Louis

Re: Question collaborative editing - Wikipedia reference

[Ergus]

This email is from my absolute ignorance about advanced networks:

On Tue, Oct 27, 2020 at 10:45:28PM +0000, Qiantan Hong wrote:
>> Just as Gobby editor, it leaves connecting issues to the
>> administrator, it does not solves it itself. It is external issue.
>
>We might be able to automate this as best effort and eliminate
>the need for any network knowledge/expertise for most users.
>I will explain below.
>
>> Sorry for coming late to the party, and for this naïve and perhaps repeated question/suggestion (I tried searching the archive but could not find any efficient or reliable way), but can some service running on Savannah or Gnu.org <http://gnu.org/> act as the central point, for those cases when the collaborators cannot easily connect by other means?'
>
>I don’t think that’s an ideal way to do it, unless we really don’t know
>how to do it by other means, for the followingreasons:
>
>- I assumes either Savannah or Gnu.org <http://gnu.org/> server is just a single server
>with a fixed geolocation on the planet. It might work well for computers
>around it, but the latency will be huge if the computer happen to be
>in the other hemisphere from our server. CDN can’t help anything here.
>
>- This will overlay all the traffic through GNU server, which is unnecessary
>in lots of cases (e.g. when it’s possible to use STUN to traverse the NATs).
>That will put maintenance burden on the server admin I think
>
>I’m roaming through relevant material recently and I think I have those
>options in mind
>
>- use libnice to do the NAT traversal job. It will be convenient if there’s
>any TCP tunneling tool built on libnice. Nobody mention any so far, so
>maybe we need to implement one in C.
>
 From my point of view (where I usually tend to reinvent the wheel). The
GNU servers could act just as "rendezvous" servers to establish the
connections between remote users.

Lets say a users1 starts a public-remote session and gets a hash, ID,
whatever. Then share that hash with the other users and they can connect
directly to his emacs. To add extra security, the user could have
another hash that is never sent to the server but used to autenticate on
his session and needs to be shared directly with his "friends".

The server is somehow a cheap process that only receives requests like
"new session -> store ip+port -> return hash (ID)" and "connect to ID
(the hash): return ip+port". With a timeout +keep-alive check and so.

After that we only need a UDP hole punching algorithm to connect
remotely directly between emacs servers p2p. This is very simple to
implement in C, the algorithms for hole punching are actually very
simple, considering the complex architectures (old and new) and with
this we can cover probably most of the networks around, complex NATs and
90% of the routers ISP providers around.

The central server does not need to know/store/log or get track of the
connections between remote session, redirect any editing information. It
just remember the sessions started or that sent a keep alive in the
last... lets say 15 minutes.

I didn't find any library to do this and I am totally out of time, I
tried some basic examples available on internet/github codes and they
seemed to work for simple application traversing the NATs...

Just as examples:

[1] https://github.com/ckennelly/hole-punch
[2] https://github.com/mwarning/UDP-hole-punching-examples


>- use ipfs p2p tunneling functionality
>https://docs.ipfs.io/reference/cli/#ipfs-p2p <https://docs.ipfs.io/reference/cli/#ipfs-p2p>
>I can immediately add some elisp to automate this such that user just issue
>the new session Emacs command, then get a accessible IPFS URI,
>if we think this is the way to go. I dislike the IPFS implementation however
>because it’s return in a UNIXer language whose name shall not be mentioned.
>Also IPFS is not a GNU project.
>
>- use Tor hidden service.
>It happens to traverse NAT (or some firewall) as a byproduct of its anonymity.
>Some drawback
>   + we also pay the anonymity tax - lower bandwidth and higher latency,
>      because of the relays between.
>   + starting hidden service require changing some config file and usually
>      require root access, which make it a bit more cumbersome to automate
>      in Elisp.
>
>- GNUnet. I wish we were running our whole Internet over it. However
>   I doubt gnunet-cadet is usable with acceptable latency right now.
>
>

Re: Question collaborative editing - tools to connect

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 2153 bytes --]

I wrote some experimental elisp (haven’t pushed but I
should finish it by this weekend) to automatically proxy
the connection over tuntox. It’s pretty fast after
the connection is established, but can take about 10s
to establish the connection even if I’m connecting from
localhost to localhost.
Also, seems that it doesn’t drop the connection to server
even if the client close the connection.
Server Emacs -1- tuntox -???- tuntox -2- Client Emacs
when TCP connection 2 closes, nothing happen to TCP 
connection 1.
Maybe this is not a big deal anyway, I can easily add a
command to kill connection manually on the server.

> On Oct 28, 2020, at 12:29 PM, Jean Louis <bugs@gnu.support> wrote:
> 
> * Qiantan Hong <qhong@mit.edu> [2020-10-26 23:37]:
>> ngrok is free software I think, both server and client
>> side. However last time I check it myself it does
>> seems that you need to sign up to use TCP tunnel
>> on their official servers and the sign up page require
>> non-free JS. That’s why I didn’t mention it in my documentation.
> 
> SSH is easiest
> 
> 
> 3 stunnel -- should be accompanied with SSH
> =========
> 
>  Beschreibung : A program that allows you to encrypt arbitrary TCP
>  connections inside SSL, with libressl and OpenRC support URL :
>  <https://www.stunnel.org/>
> 
> 
> 4 tuntox 
> ========
> 
>  Beschreibung : Tunnel TCP connections over the Tox protocol, with
>  logger recommendation and OpenRC support URL :
>  <https://github.com/gjedeer/tuntox>
> 
> 
> 1 iodine 
> ========
> 
>  Name : iodine Beschreibung : Tunnel IPv4 data through a DNS server URL
>  : <http://code.kryo.se/iodine>
> 
>  It can be used through DNS, very handy when mobile data is out of
>  credit but one has access to DNS system. Works with many
>  providers. That would be collaborative avoidance of Internet fees.
> 
> 
> 2 ptunnel
> =========
> 
>  Beschreibung : A tool for reliably tunneling TCP connections over ICMP
>  echo request and reply packets URL :
>  <http://www.cs.uit.no/~daniels/PingTunnel>
> 
>  Similar like above iodined.
> 
> 
> -- 
> Jean Louis


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1858 bytes --]

Re: Question collaborative editing - tools to connect

[Jean Louis]

* Qiantan Hong <qhong@mit.edu> [2020-10-31 07:17]:
> I wrote some experimental elisp (haven’t pushed but I
> should finish it by this weekend) to automatically proxy
> the connection over tuntox. It’s pretty fast after
> the connection is established, but can take about 10s
> to establish the connection even if I’m connecting from
> localhost to localhost.
> Also, seems that it doesn’t drop the connection to server
> even if the client close the connection.
> Server Emacs -1- tuntox -???- tuntox -2- Client Emacs
> when TCP connection 2 closes, nothing happen to TCP 
> connection 1.
> Maybe this is not a big deal anyway, I can easily add a
> command to kill connection manually on the server.

It solves it to connect peer to peer. There are few ways.

Though it does not solve SSL server problem.

Re: Question collaborative editing - crdt.el, does it use SSL?

[Robert Pluim]

Jean Louis <bugs@gnu.support> writes:

> * Qiantan Hong <qhong@mit.edu> [2020-10-29 02:09]:
>> I know that make-network-process can make TLS connection,
>> however it seems that it only works when Emacs is behaving as
>> client.
>
> You tried with these?
>
> :plist PLIST -- Install PLIST as the new process’s initial plist.
> :tls-parameters LIST -- is a list that should be supplied if you’re
> opening a TLS connection.  The first element is the TLS type (either
> ‘gnutls-x509pki’ or ‘gnutls-anon’), and the remaining elements should
> be a keyword list accepted by gnutls-boot (as returned by
> ‘gnutls-boot-parameters’).
>
> :server QLEN -- if QLEN is non-nil, create a server process for the
> specified FAMILY, SERVICE, and connection type (stream or datagram).
> If QLEN is an integer, it is used as the max. length of the server’s
> pending connection queue (also known as the backlog); the default
> queue length is 5.  Default is to create a client process.

emacs can function as a udp/tcp server, but where would it get the
certificate from to function as a TLS server?

Robert

Re: Question collaborative editing - crdt.el, does it use SSL?

[Jean Louis]

* Robert Pluim <rpluim@gmail.com> [2020-11-02 12:48]:
> Jean Louis <bugs@gnu.support> writes:
> 
> > * Qiantan Hong <qhong@mit.edu> [2020-10-29 02:09]:
> >> I know that make-network-process can make TLS connection,
> >> however it seems that it only works when Emacs is behaving as
> >> client.
> >
> > You tried with these?
> >
> > :plist PLIST -- Install PLIST as the new process’s initial plist.
> > :tls-parameters LIST -- is a list that should be supplied if you’re
> > opening a TLS connection.  The first element is the TLS type (either
> > ‘gnutls-x509pki’ or ‘gnutls-anon’), and the remaining elements should
> > be a keyword list accepted by gnutls-boot (as returned by
> > ‘gnutls-boot-parameters’).
> >
> > :server QLEN -- if QLEN is non-nil, create a server process for the
> > specified FAMILY, SERVICE, and connection type (stream or datagram).
> > If QLEN is an integer, it is used as the max. length of the server’s
> > pending connection queue (also known as the backlog); the default
> > queue length is 5.  Default is to create a client process.
> 
> emacs can function as a udp/tcp server, but where would it get the
> certificate from to function as a TLS server?

Every server for crdt.el could create self-signed certificate. I guess
there are various tools for that, like libressl, openssl (deprecated)
and there is certtool - GnuTLS certificate tool

Reference:
https://gnutls.org/manual/html_node/certtool-Invocation.html#certtool-Invocation