Collaborative editing.

Collaborative editing.

[Perry E. Metzger]

Howdy!

For years now, pair programming has been of interest to loads of 
programmers, and I've done quite a bit of it myself. For a long time, 
I'd do this with Emacs by running Screen on a machine and having myself 
and whoever I was pairing with both connect up to the same Emacs 
(running in a terminal) that way.

However, it's increasingly difficult to get the full benefits of Emacs 
in ordinary terminals; there are too many things a gui can do that a 
terminal can't.

I've also discovered that many other editors (such as VSCode and Atom) 
seem to now have dedicated modes for doing collaborative editing, and 
apparently do it quite well. Indeed, many people I asked directed me to 
VSCode for this when I asked around about methods to do it for Emacs. 
(SubEthaEdit, which is now free software, also allows such things, but 
it is a much more limited editor.)

Especially with more and more working programmers doing their jobs from 
home but wanting to work with colleagues far away, it would seem like 
having truly good support for collaborative editing baked in to Emacs by 
some means would be a good idea.

I know there have been some experiments with collaborative editing modes 
in the past that were written purely in Elisp but none seem to be 
currently maintained and I'm not sure if any were actually very good to 
begin with.

Anyone have thoughts on how one could get Emacs to be a really top 
flight collaborative editing environment, especially for programmers? 
Just to be clear, one would want both programmers to be in distant 
locations, but to get essentially the same view of the file being 
edited, the same view of any UI elements like popups, and to be able to 
control the keyboard and mouse more or less simultaneously. Presumably 
one of the two Emacsen would be the primary one and the other one just a 
remote display, though other architectures are possible.

Obvious design alternatives are dealing with some sort of quite literal 
display sharing mechanism in which a VNC-like protocol is used, a 
slightly more "semantically" based display sharing protocol in which a 
remote display is sent a series of high level commands about updates, 
some sort of more literal collaborative editing in which diffs to text 
get sent back and forth (but this would not make it easy for both 
programmers to see the same view of the text, including popups etc.), 
and there are probably other places in the design space.

Perry

Re: Collaborative editing.

[Tim Cross]

"Perry E. Metzger" <perry@piermont.com> writes:

> Howdy!
>
> For years now, pair programming has been of interest to loads of programmers,
> and I've done quite a bit of it myself. For a long time, I'd do this with Emacs
> by running Screen on a machine and having myself and whoever I was pairing with
> both connect up to the same Emacs (running in a terminal) that way.
>
> However, it's increasingly difficult to get the full benefits of Emacs in
> ordinary terminals; there are too many things a gui can do that a terminal
> can't.
>
> I've also discovered that many other editors (such as VSCode and Atom) seem to
> now have dedicated modes for doing collaborative editing, and apparently do it
> quite well. Indeed, many people I asked directed me to VSCode for this when I
> asked around about methods to do it for Emacs. (SubEthaEdit, which is now free
> software, also allows such things, but it is a much more limited editor.)
>
> Especially with more and more working programmers doing their jobs from home but
> wanting to work with colleagues far away, it would seem like having truly good
> support for collaborative editing baked in to Emacs by some means would be a
> good idea.
>
> I know there have been some experiments with collaborative editing modes in the
> past that were written purely in Elisp but none seem to be currently maintained
> and I'm not sure if any were actually very good to begin with.
>
> Anyone have thoughts on how one could get Emacs to be a really top flight
> collaborative editing environment, especially for programmers? Just to be clear,
> one would want both programmers to be in distant locations, but to get
> essentially the same view of the file being edited, the same view of any UI
> elements like popups, and to be able to control the keyboard and mouse more or
> less simultaneously. Presumably one of the two Emacsen would be the primary one
> and the other one just a remote display, though other architectures are
> possible.
>
> Obvious design alternatives are dealing with some sort of quite literal display
> sharing mechanism in which a VNC-like protocol is used, a slightly more
> "semantically" based display sharing protocol in which a remote display is sent
> a series of high level commands about updates, some sort of more literal
> collaborative editing in which diffs to text get sent back and forth (but this
> would not make it easy for both programmers to see the same view of the text,
> including popups etc.), and there are probably other places in the design space.
>

Just as another data point, I've done this similar to your screen
example (except we used tmux). We also used spacemacs, which has it's
'hybrid' mode, which supports both traditional Emacs key bindings and
Evil, which was handy because the other user was not an Emacs user, but
they were familiar with VI. It worked OK, but was definitely not as
'polished' as VSCode's remote collaboration/editing mode. The VSCode
model does work very nicely.

A remote/pair programming mode for Emacs would be nice. What would be
very nice is if it was possible for Emacs to leverage off the same
protocol as VSCode. It would be very nice if you could pair program with
someone who is using VSCode while you are using Emacs (and vice versa).
One of the limitations with the screen/tmux model is the expectation
both parties are Emacs users. It is rare I find many other Emacs users
on teams I've worked with. In the last few years, VSCode and Atom seem
to be far more common. 

Re: Collaborative editing.

[Eli Zaretskii]

> Date: Thu, 12 Aug 2021 19:43:25 -0400
> From: "Perry E. Metzger" <perry@piermont.com>
> 
> Anyone have thoughts on how one could get Emacs to be a really top 
> flight collaborative editing environment, especially for programmers? 

I suggest to search the archives of this list for "collaborative",
there were a few large discussions last year.  One of the latest
attempts to add these capabilities to Emacs is here:

  https://code.librehq.com/qhong/crdt.el/

Re: Collaborative editing.

[Jean Louis]

* Perry E. Metzger <perry@piermont.com> [2021-08-13 02:44]:
> I know there have been some experiments with collaborative editing modes in
> the past that were written purely in Elisp but none seem to be currently
> maintained and I'm not sure if any were actually very good to begin with.

CRDT works just fine and is well maintained, you can contact author
Qiantan Hong <qhong@mit.edu> at any time you wish.

Do:

$ git clone https://code.librehq.com/qhong/crdt.el.git

Let me know if you need any help or assistance to start with
collaborative editing.


-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: Collaborative editing.

[Ergus]

[-- Attachment #1: Type: text/plain, Size: 1011 bytes --]

Could we try to add crdt to Elpa? Is the author somehow opposed to do the paperwork or so?

On August 15, 2021 7:46:43 AM GMT+02:00, Jean Louis <bugs@gnu.support> wrote:
>* Perry E. Metzger <perry@piermont.com> [2021-08-13 02:44]:
>> I know there have been some experiments with collaborative editing modes in
>> the past that were written purely in Elisp but none seem to be currently
>> maintained and I'm not sure if any were actually very good to begin with.
>
>CRDT works just fine and is well maintained, you can contact author
>Qiantan Hong <qhong@mit.edu> at any time you wish.
>
>Do:
>
>$ git clone https://code.librehq.com/qhong/crdt.el.git
>
>Let me know if you need any help or assistance to start with
>collaborative editing.
>
>
>-- 
>Jean
>
>Take action in Free Software Foundation campaigns:
>https://www.fsf.org/campaigns
>
>In support of Richard M. Stallman
>https://stallmansupport.org/
>

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 1689 bytes --]

Re: Collaborative editing.

[Philip Kaludercic]

Ergus <spacibba@aol.com> writes:

> Could we try to add crdt to Elpa? Is the author somehow opposed to do the paperwork or so?

The package still seems to be on version 0.0.0, and the HACKING[0] file
indicates that a few intended items are not implemented yet. It might
make sense to push for a preliminary version to be published as to
provide a basic collaborative environment available on ELPA (or NonGNU
ELPA if necessary), and then later work on full-compatibility.

[0] https://code.librehq.com/qhong/crdt.el/-/raw/master/HACKING.org

> On August 15, 2021 7:46:43 AM GMT+02:00, Jean Louis <bugs@gnu.support> wrote:
>>* Perry E. Metzger <perry@piermont.com> [2021-08-13 02:44]:
>>> I know there have been some experiments with collaborative editing modes in
>>> the past that were written purely in Elisp but none seem to be currently
>>> maintained and I'm not sure if any were actually very good to begin with.
>>
>>CRDT works just fine and is well maintained, you can contact author
>>Qiantan Hong <qhong@mit.edu> at any time you wish.
>>
>>Do:
>>
>>$ git clone https://code.librehq.com/qhong/crdt.el.git
>>
>>Let me know if you need any help or assistance to start with
>>collaborative editing.
>>
>>
>>-- 
>>Jean
>>
>>Take action in Free Software Foundation campaigns:
>>https://www.fsf.org/campaigns
>>
>>In support of Richard M. Stallman
>>https://stallmansupport.org/
>>

-- 
	Philip Kaludercic

Re: Collaborative editing.

[Philip Kaludercic]

Ergus <spacibba@aol.com> writes:

> Could we try to add crdt to Elpa? Is the author somehow opposed to do the paperwork or so?

The package still seems to be on version 0.0.0, and the HACKING[0] file
indicates that a few intended items are not implemented yet. It might
make sense to push for a preliminary version to be published as to
provide a basic collaborative environment available on ELPA (or NonGNU
ELPA if necessary), and then later work on full-compatibility.

[0] https://code.librehq.com/qhong/crdt.el/-/raw/master/HACKING.org

> On August 15, 2021 7:46:43 AM GMT+02:00, Jean Louis <bugs@gnu.support> wrote:
>>* Perry E. Metzger <perry@piermont.com> [2021-08-13 02:44]:
>>> I know there have been some experiments with collaborative editing modes in
>>> the past that were written purely in Elisp but none seem to be currently
>>> maintained and I'm not sure if any were actually very good to begin with.
>>
>>CRDT works just fine and is well maintained, you can contact author
>>Qiantan Hong <qhong@mit.edu> at any time you wish.
>>
>>Do:
>>
>>$ git clone https://code.librehq.com/qhong/crdt.el.git
>>
>>Let me know if you need any help or assistance to start with
>>collaborative editing.
>>
>>
>>-- 
>>Jean
>>
>>Take action in Free Software Foundation campaigns:
>>https://www.fsf.org/campaigns
>>
>>In support of Richard M. Stallman
>>https://stallmansupport.org/
>>

-- 
	Philip Kaludercic

Re: Collaborative editing.

[Ergus]

On Thu, Aug 19, 2021 at 09:33:28AM +0000, Philip Kaludercic wrote:
>Ergus <spacibba@aol.com> writes:
>
>> Could we try to add crdt to Elpa? Is the author somehow opposed to do the paperwork or so?
>
>The package still seems to be on version 0.0.0, and the HACKING[0] file
>indicates that a few intended items are not implemented yet. It might

>make sense to push for a preliminary version to be published as to
>provide a basic collaborative environment available on ELPA (or NonGNU
>ELPA if necessary), and then later work on full-compatibility.

This is the point. When some users know about the package searching in
the packages-list; maybe they will want to collaborate or report issues,
so it won't becomes a single man effort. IMHO a package doesn't really
exist until it is in Elpa or at least Melpa.

Otherwise in a couple of years there will be someone starting again
another similar effort from scratch.
>
>[0] https://code.librehq.com/qhong/crdt.el/-/raw/master/HACKING.org
>
>> On August 15, 2021 7:46:43 AM GMT+02:00, Jean Louis <bugs@gnu.support> wrote:
>>>* Perry E. Metzger <perry@piermont.com> [2021-08-13 02:44]:
>>>> I know there have been some experiments with collaborative editing modes in
>>>> the past that were written purely in Elisp but none seem to be currently
>>>> maintained and I'm not sure if any were actually very good to begin with.
>>>
>>>CRDT works just fine and is well maintained, you can contact author
>>>Qiantan Hong <qhong@mit.edu> at any time you wish.
>>>
>>>Do:
>>>
>>>$ git clone https://code.librehq.com/qhong/crdt.el.git
>>>
>>>Let me know if you need any help or assistance to start with
>>>collaborative editing.
>>>
>>>
>>>--
>>>Jean
>>>
>>>Take action in Free Software Foundation campaigns:
>>>https://www.fsf.org/campaigns
>>>
>>>In support of Richard M. Stallman
>>>https://stallmansupport.org/
>>>
>
>-- 
>	Philip Kaludercic

Re: Collaborative editing.

[dick]

E> Otherwise in a couple of years there will be someone starting again another
E> similar effort from scratch.

I wouldn't worry about that.  Google remains the finder of lost children *par
excellence*.  If a project attained any kind of critical mass (such as
crdt.el), Google will find it, not the nugatory search capabilities of the ELPAs.

Re: Collaborative editing.

[Perry E. Metzger]

On 8/19/21 10:18, Ergus wrote:
>
> This is the point. When some users know about the package searching in
> the packages-list; maybe they will want to collaborate or report issues,
> so it won't becomes a single man effort. IMHO a package doesn't really
> exist until it is in Elpa or at least Melpa.
>
> Otherwise in a couple of years there will be someone starting again
> another similar effort from scratch.
>
I'd go further and say that like tramp, gnus, or org mode, or other 
really critical features, collaborative editing probably should be part 
of the base distribution.

Perry

Re: Collaborative editing.

[Philip Kaludercic]

Ergus <spacibba@aol.com> writes:

> On Thu, Aug 19, 2021 at 09:33:28AM +0000, Philip Kaludercic wrote:
>>Ergus <spacibba@aol.com> writes:
>>
>>> Could we try to add crdt to Elpa? Is the author somehow opposed to do the paperwork or so?
>>
>>The package still seems to be on version 0.0.0, and the HACKING[0] file
>>indicates that a few intended items are not implemented yet. It might
>
>>make sense to push for a preliminary version to be published as to
>>provide a basic collaborative environment available on ELPA (or NonGNU
>>ELPA if necessary), and then later work on full-compatibility.
>
> This is the point. When some users know about the package searching in
> the packages-list; maybe they will want to collaborate or report issues,
> so it won't becomes a single man effort. IMHO a package doesn't really
> exist until it is in Elpa or at least Melpa.

I agree, though in this case there is also the issue of having a package
that is hard to debug and test on your own, because the real bugs will
probably only pop up in hard to replicate configurations
(transcontinental-collaboration, obscure network configurations, etc.)

> Otherwise in a couple of years there will be someone starting again
> another similar effort from scratch.

I'd be intersted to see what Qiantan has to say about all of this. It
seems like he changed his email address according to the last commit in
the repository, so I update the CC'ed address in this thread too in case
he is missing out on the conversation.

>>
>>[0] https://code.librehq.com/qhong/crdt.el/-/raw/master/HACKING.org
>>
>>> On August 15, 2021 7:46:43 AM GMT+02:00, Jean Louis <bugs@gnu.support> wrote:
>>>>* Perry E. Metzger <perry@piermont.com> [2021-08-13 02:44]:
>>>>> I know there have been some experiments with collaborative editing modes in
>>>>> the past that were written purely in Elisp but none seem to be currently
>>>>> maintained and I'm not sure if any were actually very good to begin with.
>>>>
>>>>CRDT works just fine and is well maintained, you can contact author
>>>>Qiantan Hong <qhong@mit.edu> at any time you wish.
>>>>
>>>>Do:
>>>>
>>>>$ git clone https://code.librehq.com/qhong/crdt.el.git
>>>>
>>>>Let me know if you need any help or assistance to start with
>>>>collaborative editing.
>>>>
>>>>
>>>>--
>>>>Jean
>>>>
>>>>Take action in Free Software Foundation campaigns:
>>>>https://www.fsf.org/campaigns
>>>>
>>>>In support of Richard M. Stallman
>>>>https://stallmansupport.org/
>>>>
>>
>> -- 	Philip Kaludercic
>

-- 
	Philip Kaludercic

Re: Collaborative editing.

[Eli Zaretskii]

> Date: Thu, 19 Aug 2021 11:19:54 -0400
> From: "Perry E. Metzger" <perry@piermont.com>
> 
> I'd go further and say that like tramp, gnus, or org mode, or other 
> really critical features, collaborative editing probably should be part 
> of the base distribution.

I agree, provided that we can find a package which implements that in
a clean enough manner, and the author agrees to contribute it.

Re: Collaborative editing.

[Qiantan Hong]

[-- Attachment #1: Type: text/plain, Size: 1562 bytes --]


Hi guys!

Sorry I've been busy lately and haven't seen this till now.

>>> Could we try to add crdt to Elpa? Is the author somehow opposed to do the paperwork or so?
I've already done the paperwork! Our school's office was quite slow but it's done now.

>>
>>The package still seems to be on version 0.0.0, and the HACKING[0] file
>>indicates that a few intended items are not implemented yet. It might
>
>>make sense to push for a preliminary version to be published as to
>>provide a basic collaborative environment available on ELPA (or NonGNU
>>ELPA if necessary), and then later work on full-compatibility.
Those few intended items was in fact rather very ambitious plan,
and not necessarily needed for "usual" collaboration experience.
They're suppose to enable support for, say, sharing a XScheme.el
buffer with an active running process so people can share a Lisp image
through Emacs.

Basically all functionalities for "usual" collaboration editing are all already implemented.

I'm happy to release crdt.el into either ELPA, or into mainstream Emacs
(maybe after it's battle tested on ELPA first)!

There are indeed issue about the difficulty in debugging, there is an open issue
on librehq that I still haven't reproduced yet.
It may help if we have a larger testing base.
I think it will also be very helpful if some other hackers get to understand
the internal of crdt.el and could hack when they happen to trigger some bugs,
as debugging another user's Emacs via carbon-based bio-SSH is hard.

Best,
Qiantan



[-- Attachment #2: Type: text/html, Size: 8592 bytes --]

Re: Collaborative editing.

[Qiantan Hong]

Hi guys!

Sorry I've been busy lately and haven't seen this till now.

>>> Could we try to add crdt to Elpa? Is the author somehow opposed to do the paperwork or so?
I've already done the paperwork! Our school's office was quite slow but it's done now.

>>
>>The package still seems to be on version 0.0.0, and the HACKING[0] file
>>indicates that a few intended items are not implemented yet. It might
>
>>make sense to push for a preliminary version to be published as to
>>provide a basic collaborative environment available on ELPA (or NonGNU
>>ELPA if necessary), and then later work on full-compatibility.
Those few intended items was in fact rather very ambitious plan,
and not necessarily needed for "usual" collaboration experience.
They're suppose to enable support for, say, sharing a XScheme.el
buffer with an active running process so people can share a Lisp image
through Emacs.

Basically all functionalities for "usual" collaboration editing are all already implemented.

I'm happy to release crdt.el into either ELPA, or into mainstream Emacs
(maybe after it's battle tested on ELPA first)!

There are indeed issue about the difficulty in debugging, there is an open issue
on librehq that I still haven't reproduced yet.
It may help if we have a larger testing base.
I think it will also be very helpful if some other hackers get to understand
the internal of crdt.el and could hack when they happen to trigger some bugs,
as debugging another user's Emacs via carbon-based bio-SSH is hard.

Best,
Qiantan

Re: Collaborative editing.

[Philip Kaludercic]

Qiantan Hong <qhong@alum.mit.edu> writes:

> I'm happy to release crdt.el into either ELPA, or into mainstream Emacs
> (maybe after it's battle tested on ELPA first)!

I think adding it to ELPA first would be preferable. Just note that you
would have to update the copyright notice before it could be added, and
I think incrementing the version from 0.0.0 would be nice too.

-- 
	Philip Kaludercic

Re: Collaborative editing.

[Qiantan Hong]

Sounds good, I’ve updated the copyright notice and assigned version number 0.1.0

> On Aug 28, 2021, at 4:40 AM, Philip Kaludercic <philipk@posteo.net> wrote:
> 
> Qiantan Hong <qhong@alum.mit.edu> writes:
> 
>> I'm happy to release crdt.el into either ELPA, or into mainstream Emacs
>> (maybe after it's battle tested on ELPA first)!
> 
> I think adding it to ELPA first would be preferable. Just note that you
> would have to update the copyright notice before it could be added, and
> I think incrementing the version from 0.0.0 would be nice too.
> 
> -- 
> 	Philip Kaludercic

Re: Collaborative editing.

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 216 bytes --]

Qiantan Hong <qhong@mit.edu> writes:

> Sounds good, I’ve updated the copyright notice and assigned version number 0.1.0

Yep, I managed to build the package now, using the following package
specification:


[-- Attachment #2: Type: text/plain, Size: 551 bytes --]

diff --git a/elpa-packages b/elpa-packages
index 8eab656624..576c96e704 100644
--- a/elpa-packages
+++ b/elpa-packages
@@ -105,6 +105,7 @@
  ("counsel"		:url "https://github.com/abo-abo/swiper")
  ("cpio-mode"		:url "https://github.com/dlewan/cpio-mode")
  ("crisp"		:url nil)
+ ("crdt"		:url "https://code.librehq.com/qhong/crdt.el")
  ;; ("crossword"		:url "https://github.com/Boruch-Baum/emacs-crossword")
  ;; ("csharp-mode"		:url "https://github.com/emacs-csharp/csharp-mode")
  ("csharp-mode"		:url "https://github.com/emacs-csharp/csharp-mode"

[-- Attachment #3: Type: text/plain, Size: 24 bytes --]


-- 
	Philip Kaludercic

Re: Collaborative editing.

[Ag Ibragimov]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Thu, 12 Aug 2021 19:43:25 -0400
>> From: "Perry E. Metzger" <perry@piermont.com>
>> 
>> Anyone have thoughts on how one could get Emacs to be a really top 
>> flight collaborative editing environment, especially for programmers? 
>
> I suggest to search the archives of this list for "collaborative",
> there were a few large discussions last year.  One of the latest
> attempts to add these capabilities to Emacs is here:
>
>   https://code.librehq.com/qhong/crdt.el/

Sorry for so nonchalantly changing the subject, but speaking of
searching the archives. Since not too long ago I've been having trouble
searching using "field-specified searching", i.e., using the +subject, or
+from, and most importantly - using +message-id operator.

I have a function that allows me to find the current email thread in the archives
using email's message-id; basically, it calls
(notmuch-show-get-message-id); then launches https://lists.gnu.org
and tries to search with a search string like this:

  +message-id:<YRiqQ7auhgw8LW0z@protected.localdomain>

And it used to work, but at some point, it stopped working, and that's
annoying. I've been helplessly shaking my fist whenever I try to find a thread
in the archives, but other than that, I have no idea how to cope with
this.

Does anyone know what's changed, why the search doesn't work as before anymore?

Re: Search for message-id (was: Collaborative editing.)

[Eli Zaretskii]

> From: Ag Ibragimov <agzam.ibragimov@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 01 Sep 2021 01:32:40 -0500
> 
> Sorry for so nonchalantly changing the subject

Please in the future actually change the Subject when you do so.  (I
did it now.)

> but speaking of
> searching the archives. Since not too long ago I've been having trouble
> searching using "field-specified searching", i.e., using the +subject, or
> +from, and most importantly - using +message-id operator.
> 
> I have a function that allows me to find the current email thread in the archives
> using email's message-id; basically, it calls
> (notmuch-show-get-message-id); then launches https://lists.gnu.org
> and tries to search with a search string like this:
> 
>   +message-id:<YRiqQ7auhgw8LW0z@protected.localdomain>
> 
> And it used to work, but at some point, it stopped working, and that's
> annoying. I've been helplessly shaking my fist whenever I try to find a thread
> in the archives, but other than that, I have no idea how to cope with
> this.
> 
> Does anyone know what's changed, why the search doesn't work as before anymore?

My guess is that this has something to do with the changes done in
configuration of mailman-managed GNU mailing lists to avoid the DKIM
lunacy, which I think also gave us the "[External]" prefixes and
mangled From addresses.

But the right place to ask this is not here, it is by sending them to
these two addresses:

  mailman@gnu.org
  sysadmin@gnu.org