Buffers with buffers (transclusion)

Buffers with buffers (transclusion)

[Vladimir Kazanov]

Hi all,

Recently I got interested in the best way to share text between buffers.

This is very useful, and was discussed a bunch of times in the past
(https://lists.gnu.org/archive/html/emacs-devel/2020-04/msg00141.html,
https://lists.gnu.org/archive/html/emacs-devel/2018-07/msg00863.html).
There's also a nice overview of these discussions on Github:
https://github.com/alphapapa/transclusion-in-emacs.

My personal interest was sparkled by a prototype of a tool I published
recently (https://github.com/vkazanov/chunk-edit) that makes it
possible to edit multiple buffers from within a single host one.

I would love some advice on what approach Emacs maintainers might be
interested in. Here's are the ideas I have going from easy to hard:

1. Just track and copy all buffer edits using, say, overlay hooks.
This is what I did in chunk-edit.el. The problem with this approach is
inefficiency and brittleness.

2. Introduce indirection on the deepest possible level, e.g., come up
with something like composite buffers (buffers consisting of multiple
buffers), or make buffers use segments internally.

3. Make the iterator in xdisp.c understand a new display spec -
(display . buffer), which would make it possible to display external
buffers within the current buffer. From what I see right now (xdisp.c
is not the easiest piece of code), this would make the text not
editable and is not the same use-case I had in mind but might open up
a bunch of  interesting capabilities.

Did anybody look into something like this recently? Any
comments/ideas/guidance would be great.

--
Regards,

Vladimir Kazanov

Re: Buffers with buffers (transclusion)

[Eli Zaretskii]

> From: Vladimir Kazanov <vekazanov@gmail.com>
> Date: Tue, 15 Oct 2024 14:53:44 +0100
> 
> Recently I got interested in the best way to share text between buffers.
> 
> This is very useful, and was discussed a bunch of times in the past
> (https://lists.gnu.org/archive/html/emacs-devel/2020-04/msg00141.html,
> https://lists.gnu.org/archive/html/emacs-devel/2018-07/msg00863.html).
> There's also a nice overview of these discussions on Github:
> https://github.com/alphapapa/transclusion-in-emacs.
> 
> My personal interest was sparkled by a prototype of a tool I published
> recently (https://github.com/vkazanov/chunk-edit) that makes it
> possible to edit multiple buffers from within a single host one.
> 
> I would love some advice on what approach Emacs maintainers might be
> interested in. Here's are the ideas I have going from easy to hard:

Thanks, but it is hard to reason about the ideas and possible
implementations without knowing the intended functionalities and the
use cases this should support.  So my suggestion is to describe these
first.

Re: Buffers with buffers (transclusion)

[Vladimir Kazanov]

Hi Eli,

> Thanks, but it is hard to reason about the ideas and possible
> implementations without knowing the intended functionalities and the
> use cases this should support.  So my suggestion is to describe these
> first.

Sure!

The core use case is being able to edit many files from within a single buffer.

This is what chunk-edit.el does: it makes it possible to mark regions
of interest in various buffers, which are then exposed in a temporary
buffer (*chunk-edit*). The user can then focus on editing interesting
bits of code within a single buffer.

Here's an example user story: a programmer goes through 30k lines of
code of xdisp.c and dispextern.c and finds the key bits of code he
needs to change. Instead of jumping between points of interest, or
setting up a complex window configuration, he just copies everything
into *chunk-edit* where he can safely edit all of these at once.

Secondary to this is having a reasonable way to edit these diverse
chunks of code with the comfort of syntax highlighting.

Ideally, I would like to solve both problems in one go: just *include*
a subset of a buffer within *chunk-edit*, complete with text
properties and all.

Thanks,
Vlad

--
Regards,

Vladimir Kazanov

Re: Buffers with buffers (transclusion)

[James Thomas]

Vladimir Kazanov wrote:

> Hi Eli,
>
>> Thanks, but it is hard to reason about the ideas and possible
>> implementations without knowing the intended functionalities and the
>> use cases this should support.  So my suggestion is to describe these
>> first.
>
> Sure!
>
> The core use case is being able to edit many files from within a single buffer.
>
> This is what chunk-edit.el does: it makes it possible to mark regions
> of interest in various buffers, which are then exposed in a temporary
> buffer (*chunk-edit*). The user can then focus on editing interesting
> bits of code within a single buffer.
>
> Here's an example user story: a programmer goes through 30k lines of
> code of xdisp.c and dispextern.c and finds the key bits of code he
> needs to change. Instead of jumping between points of interest, or
> setting up a complex window configuration, he just copies everything
> into *chunk-edit* where he can safely edit all of these at once.
>
> Secondary to this is having a reasonable way to edit these diverse
> chunks of code with the comfort of syntax highlighting.
>
> Ideally, I would like to solve both problems in one go: just *include*
> a subset of a buffer within *chunk-edit*, complete with text
> properties and all.

Dumb question:

How about a way to stack windows on top of each other, without any
modelines, and move to the bottom window when scrolling off the top one?

Regards,
James

Re: Buffers with buffers (transclusion)

[Eli Zaretskii]

> From: James Thomas <jimjoe@gmx.net>
> Cc: emacs-devel@gnu.org
> Date: Thu, 17 Oct 2024 04:14:38 +0530
> 
> Vladimir Kazanov wrote:
> 
> > The core use case is being able to edit many files from within a single buffer.
> >
> > This is what chunk-edit.el does: it makes it possible to mark regions
> > of interest in various buffers, which are then exposed in a temporary
> > buffer (*chunk-edit*). The user can then focus on editing interesting
> > bits of code within a single buffer.
> >
> > Here's an example user story: a programmer goes through 30k lines of
> > code of xdisp.c and dispextern.c and finds the key bits of code he
> > needs to change. Instead of jumping between points of interest, or
> > setting up a complex window configuration, he just copies everything
> > into *chunk-edit* where he can safely edit all of these at once.
> >
> > Secondary to this is having a reasonable way to edit these diverse
> > chunks of code with the comfort of syntax highlighting.
> >
> > Ideally, I would like to solve both problems in one go: just *include*
> > a subset of a buffer within *chunk-edit*, complete with text
> > properties and all.
> 
> Dumb question:
> 
> How about a way to stack windows on top of each other, without any
> modelines, and move to the bottom window when scrolling off the top one?

Yes, that's exactly the kind of thoughts I had when reading the
description.  Why do we need to display text from different sources in
the same buffer, when we already have side-by-side display in adjacent
windows?  If the problem is the decoration which set windows apart, it
is a minor issue to begin with, and we can make the separating
decorations less prominent.

Re: Buffers with buffers (transclusion)

[Vladimir Kazanov]

Eli, James,

> Yes, that's exactly the kind of thoughts I had when reading the
> description.  Why do we need to display text from different sources in
> the same buffer, when we already have side-by-side display in adjacent
> windows?  If the problem is the decoration which set windows apart, it
> is a minor issue to begin with, and we can make the separating
> decorations less prominent.

True, having a bunch of stacked windows, or  side by side windows,
works to an extent, and this is how I do it now. There are downsides
to this approach.

Firstly, all of the editing always happens in one of the windows, not
across windows: search/replace, occur, grep. Having things in one
buffer makes these things as easy as editing a single small file.

Also, the number of windows that can be comfortably used in this
manner is limited. 2? 3? 4?  Doing this within a single window is just
simpler.

Then, there's a performance question. One thing is to work on a 1-2k
LOC, another is to jump between buffers potentially having tens of
thousands of LOCs.

-- 
Regards,

Vladimir Kazanov

Re: Buffers with buffers (transclusion)

[Eli Zaretskii]

> From: Vladimir Kazanov <vekazanov@gmail.com>
> Date: Thu, 17 Oct 2024 09:38:06 +0100
> Cc: James Thomas <jimjoe@gmx.net>, emacs-devel@gnu.org
> 
> Firstly, all of the editing always happens in one of the windows, not
> across windows: search/replace, occur, grep. Having things in one
> buffer makes these things as easy as editing a single small file.

If the problem is to have some editing commands affect several
buffers, we could implement such a feature without bringing all the
text into a single buffer.  I'm guessing implementing such a feature
will be easier and will have fewer conceptual problems than splicing
text from different sources.

> Also, the number of windows that can be comfortably used in this
> manner is limited. 2? 3? 4?

With the displays we have nowadays, I'd say even 10 is not a problem,
and probably more.

What are the use cases where you need more separate sources?

> Doing this within a single window is just simpler.

It isn't, IMO.  It raises several conceptual problems that are hard to
solve.  E.g., if you type text between the end of SRC1 and the
beginning of SRC2, to which source is this added?  Or how do you
handle font-lock?  Or what do you do if each source has its own major
mode?  And there are other similar issues, solving which will be
difficult even conceptually.

> Then, there's a performance question. One thing is to work on a 1-2k
> LOC, another is to jump between buffers potentially having tens of
> thousands of LOCs.

Why do you envision a performance problem here?

Re: Buffers with buffers (transclusion)

[Vladimir Kazanov]

> > Firstly, all of the editing always happens in one of the windows, not
> > across windows: search/replace, occur, grep. Having things in one
> > buffer makes these things as easy as editing a single small file.
>
> If the problem is to have some editing commands affect several
> buffers, we could implement such a feature without bringing all the
> text into a single buffer.  I'm guessing implementing such a feature
> will be easier and will have fewer conceptual problems than splicing
> text from different sources.


> With the displays we have nowadays, I'd say even 10 is not a problem,
> and probably more.

Honestly speaking, I can't imagine a realistic way of manipulating 10
windows in a uniform way. There's always some kind of a current
window, both the way things are in the Emacs C core, and also from a
UI perspective. Maybe that's just my setup though.

Or do you mean an additional flavour of windows? Like, a host window
(group) owning subwindows, with all window decoration removed? And a
set of commands working on this set of subwindows?

> What are the use cases where you need more separate sources?
>
> > Doing this within a single window is just simpler.
>
> It isn't, IMO.  It raises several conceptual problems that are hard to
> solve.  E.g., if you type text between the end of SRC1 and the
> beginning of SRC2, to which source is this added?  Or how do you
> handle font-lock?  Or what do you do if each source has its own major
> mode?  And there are other similar issues, solving which will be
> difficult even conceptually.

Yes, corner cases are complex.

But Emacs does handle these problems, say, in overlays.

Polymode mangles major modes in a way that makes things usable with
multiple major modes imitated within a host buffer. In fact, solving
polymode's problems, or org-mode's src block problems, is what led me
to igniting this discussion.

Here's how this happens now in Polymode: an indirect buffer is
created, which gets a major mode, and then the mode's state is copied
over to the host buffer. I think it somehow handles keymaps as well
but I'd have to take a closer look.

Org-mode does something similar in src blocks: the code in question is
copied into a temporary buffer where the major mode specified by the
user is turned on, then text properties are copied over to the host
org buffer. Keymaps are not managed in any way.

> > Then, there's a performance question. One thing is to work on a 1-2k
> > LOC, another is to jump between buffers potentially having tens of
> > thousands of LOCs.
>
> Why do you envision a performance problem here?

I do not envision it, I feel it :-) Right now I am browsing
xdisp.c/buffer.c and a bunch of other related buffers. Flymake, eglot,
tree sitter all are just impossible to use as every time I switch to
the next window all hell breaks loose. So I just switch things off
completely. Sure, that's not directly related to the topic but
somewhat relevant for me anyway.

By the way, after reading the depths of xdisp.c's handling of display
and buffer.c's exposure of the underlying chunks of memory I now
wonder if it's easier to stream atomic changes between buffers? Say,
given one region in buffer 1 and one region in buffer 2 enforce a sync
of state (text and properties) between both regions.

-- 
Regards,

Vladimir Kazanov

Re: Buffers with buffers (transclusion)

[Eli Zaretskii]

> From: Vladimir Kazanov <vekazanov@gmail.com>
> Date: Thu, 17 Oct 2024 12:38:03 +0100
> Cc: jimjoe@gmx.net, emacs-devel@gnu.org
> 
> > With the displays we have nowadays, I'd say even 10 is not a problem,
> > and probably more.
> 
> Honestly speaking, I can't imagine a realistic way of manipulating 10
> windows in a uniform way.

One way to do that is using the windmove package, see
windmove-default-keybindings.  And I can easily envision other similar
ways.  In a nutshell, we can have commands to move between windows
bound to very simple-to-type keys.

> Or do you mean an additional flavour of windows? Like, a host window
> (group) owning subwindows, with all window decoration removed? And a
> set of commands working on this set of subwindows?

No, I mean basically the normal windows we have.

> > What are the use cases where you need more separate sources?
> >
> > > Doing this within a single window is just simpler.
> >
> > It isn't, IMO.  It raises several conceptual problems that are hard to
> > solve.  E.g., if you type text between the end of SRC1 and the
> > beginning of SRC2, to which source is this added?  Or how do you
> > handle font-lock?  Or what do you do if each source has its own major
> > mode?  And there are other similar issues, solving which will be
> > difficult even conceptually.
> 
> Yes, corner cases are complex.

These are not corner cases.  These are issues that pop up all over the
place as soon as you bring together text from separate sources.
Without a satisfactory solution for them, this feature will be only a
magnet for bug reports.

> > > Then, there's a performance question. One thing is to work on a 1-2k
> > > LOC, another is to jump between buffers potentially having tens of
> > > thousands of LOCs.
> >
> > Why do you envision a performance problem here?
> 
> I do not envision it, I feel it :-) Right now I am browsing
> xdisp.c/buffer.c and a bunch of other related buffers. Flymake, eglot,
> tree sitter all are just impossible to use as every time I switch to
> the next window all hell breaks loose. So I just switch things off
> completely.

tree-sitter is per buffer and handled locally, so I don't think it has
anything to do with switching buffers.  Eglot and Flymake are problems
that go right back to the conceptual difficulties I mentioned.  E.g.,
how do you use Eglot in a buffer that is not under a particular major
mode?  Same with Flymake.  I don't see how bringing text into a single
buffer could help here; I do see how it could break things.

> By the way, after reading the depths of xdisp.c's handling of display
> and buffer.c's exposure of the underlying chunks of memory I now
> wonder if it's easier to stream atomic changes between buffers? Say,
> given one region in buffer 1 and one region in buffer 2 enforce a sync
> of state (text and properties) between both regions.

Making the display engine show several buffers is actually very easy,
technically.  By contrast, the problems I mentioned above are much
harder, especially since it is not clear conceptually what the
solutions should be.

Re: Buffers with buffers (transclusion)

[Vladimir Kazanov]

Eli,

Thanks, almost convinced and this might be the last question on the
topic I have.

Either way, I'll summarize the discussion, my findings and conclusions
in an email that would wrap up the thread - just for future reference.

> Making the display engine show several buffers is actually very easy,
> technically.  By contrast, the problems I mentioned above are much
> harder, especially since it is not clear conceptually what the
> solutions should be.

So here's the final idea I had. Let's say we introduce a special kind
of relationship between pairs of buffers - a "sync pipe". An example:

(create-sync-pipe left-buffer right-buffer left-beg &optional left-end
right-beg right-end properties bidirectional)

This creates a connection that pushes all changes in one buffer to the
other, and vice versa. Might be limited to certains buffer regions
using (LEFT-BEG LEFT-END), (RIGHT-END, RIGHT-END) regions. Setting
PROPERTIES to (t) would also sync text properties, and BIDIRECTIONAL
means that changes go both ways.

The function will return a special C-level object that can be deleted
with (delete-pipe pipe).

PROPERTIES can be either t, nil or a a list of properties to sync. In
fact, being able to efficiently sync text and properties between
region is already 70% to what I had in mind.

Would you be open to reviewing such a patch?

Thank you

-- 
Regards,

Vladimir Kazanov

Re: Buffers with buffers (transclusion)

[Eli Zaretskii]

> From: Vladimir Kazanov <vekazanov@gmail.com>
> Date: Thu, 17 Oct 2024 14:24:10 +0100
> Cc: jimjoe@gmx.net, emacs-devel@gnu.org
> 
> So here's the final idea I had. Let's say we introduce a special kind
> of relationship between pairs of buffers - a "sync pipe". An example:
> 
> (create-sync-pipe left-buffer right-buffer left-beg &optional left-end
> right-beg right-end properties bidirectional)
> 
> This creates a connection that pushes all changes in one buffer to the
> other, and vice versa. Might be limited to certains buffer regions
> using (LEFT-BEG LEFT-END), (RIGHT-END, RIGHT-END) regions. Setting
> PROPERTIES to (t) would also sync text properties, and BIDIRECTIONAL
> means that changes go both ways.
> 
> The function will return a special C-level object that can be deleted
> with (delete-pipe pipe).
> 
> PROPERTIES can be either t, nil or a a list of properties to sync. In
> fact, being able to efficiently sync text and properties between
> region is already 70% to what I had in mind.
> 
> Would you be open to reviewing such a patch?

I need to know more about the implementation ideas: how will this be
implemented on the C level?  Do you envision something like
after-change-functions, which will modify the other buffer?  If so,
which portions of a buffer will have their changes reflected in the
other buffer?  Also, how will this help in the use cases you have in
mind?

Re: Buffers with buffers (transclusion)

[Suhail Singh]

Eli Zaretskii <eliz@gnu.org> writes:

> E.g., if you type text between the end of SRC1 and the beginning of
> SRC2, to which source is this added?

Couldn't this be obviated by there being a read-only substrate, embedded
within which are editable regions corresponding to SRC1 and SRC2?  I.e.,
SRC1 and SRC2 are separated by a region of text that cannot be modified.

> Or how do you handle font-lock?

Org mode has a somewhat "working" solution.  While it's not free of
corner cases, it's not clear that it would be impossible to resolve said
cases.

> Or what do you do if each source has its own major mode?

See above.

> And there are other similar issues, solving which will be difficult
> even conceptually.

Out of curiosity, is the difficulty in "other similar issues" inherent
to the concept of having a "meta" buffer within which exist multiple
regions each corresponding to a distinct buffer, or does it have more to
do with the technical details of the current implementation?

From my perspective, the value of a single such "meta" buffer is purely
in the UI.  Specifically, for me the value isn't potential performance
improvements while editing, but rather the potential improvement in
usability when performing, say, somewhat complex refactoring across a
code base.  I.e., it is entirely okay if in the background there are
distinct buffers that remain open where modifications occur, as long as
the changes are reflected into the "meta" buffer.

-- 
Suhail

Re: Buffers with buffers (transclusion)

[Eli Zaretskii]

> From: Suhail Singh <suhailsingh247@gmail.com>
> Cc: Vladimir Kazanov <vekazanov@gmail.com>,  jimjoe@gmx.net,
>   emacs-devel@gnu.org
> Date: Thu, 17 Oct 2024 10:36:22 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > E.g., if you type text between the end of SRC1 and the beginning of
> > SRC2, to which source is this added?
> 
> Couldn't this be obviated by there being a read-only substrate, embedded
> within which are editable regions corresponding to SRC1 and SRC2?  I.e.,
> SRC1 and SRC2 are separated by a region of text that cannot be modified.

I don't see how will that help, nor how it will resolve the
difficulties I was describing.

> Out of curiosity, is the difficulty in "other similar issues" inherent
> to the concept of having a "meta" buffer within which exist multiple
> regions each corresponding to a distinct buffer, or does it have more to
> do with the technical details of the current implementation?

The difficulties are conceptual at this point.  Whether they are
technical will become apparent when we agree on the concepts.
(Personally, I don't think there will be technical difficulties.)

> From my perspective, the value of a single such "meta" buffer is purely
> in the UI.

Then using adjacent windows solves that, IMO.

Re: Buffers with buffers (transclusion)

[Suhail Singh]

Eli Zaretskii <eliz@gnu.org> writes:

>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > E.g., if you type text between the end of SRC1 and the beginning of
>> > SRC2, to which source is this added?
>> 
>> Couldn't this be obviated by there being a read-only substrate, embedded
>> within which are editable regions corresponding to SRC1 and SRC2?  I.e.,
>> SRC1 and SRC2 are separated by a region of text that cannot be modified.
>
> I don't see how will that help, nor how it will resolve the
> difficulties I was describing.

It would resolve the difficulties by disallowing the premise.  I.e., one
wouldn't be allowed to "type text between the end of SRC1 and the
beginning of SRC2".

>> From my perspective, the value of a single such "meta" buffer is purely
>> in the UI.
>
> Then using adjacent windows solves that, IMO.

It's not clear that when you say "solves that", that we mean the same
thing by "that".  Consider all the buffer navigation and modification
commands.  If a desired criterion is for the UI to not require the user
to remember alternate commands/keybindings, how would that work?  For
instance, the idea would be for M-x occur in the "meta" buffer to do
something like multi-occur.  How would this work with adjacent windows?

On a related note, part of the buffer-in-buffer idea (to me) implies a
way of specifying buffers (and regions therein) of interest.  The
challenge being in making said specification (somewhat) stable across
revisions.  I.e., naive linking via absolute line numbers wouldn't be
sufficient.

-- 
Suhail

Re: Buffers with buffers (transclusion)

[Eli Zaretskii]

> From: Suhail Singh <suhailsingh247@gmail.com>
> Cc: Suhail Singh <suhailsingh247@gmail.com>,  vekazanov@gmail.com,
>   jimjoe@gmx.net,  emacs-devel@gnu.org
> Date: Thu, 17 Oct 2024 12:28:30 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> > E.g., if you type text between the end of SRC1 and the beginning of
> >> > SRC2, to which source is this added?
> >> 
> >> Couldn't this be obviated by there being a read-only substrate, embedded
> >> within which are editable regions corresponding to SRC1 and SRC2?  I.e.,
> >> SRC1 and SRC2 are separated by a region of text that cannot be modified.
> >
> > I don't see how will that help, nor how it will resolve the
> > difficulties I was describing.
> 
> It would resolve the difficulties by disallowing the premise.  I.e., one
> wouldn't be allowed to "type text between the end of SRC1 and the
> beginning of SRC2".

That is already a possible solution for the conceptual problem I
described.  And I'm not sure such a solution is reasonable for Emacs.

> >> From my perspective, the value of a single such "meta" buffer is purely
> >> in the UI.
> >
> > Then using adjacent windows solves that, IMO.
> 
> It's not clear that when you say "solves that", that we mean the same
> thing by "that".  Consider all the buffer navigation and modification
> commands.  If a desired criterion is for the UI to not require the user
> to remember alternate commands/keybindings, how would that work?

One possibility is to have a minor mode where C-n at the last visible
line of a window will switch to the window below.  And similarly with
C-f and C-b.

IOW, there could be an infinite number of possible solutions for the
ease of moving between the different portions of the text, which do
not require them to be in the same buffer.

> For instance, the idea would be for M-x occur in the "meta" buffer
> to do something like multi-occur.  How would this work with adjacent
> windows?

See above (if multi-occur for some reason is not enough already).

> On a related note, part of the buffer-in-buffer idea (to me) implies a
> way of specifying buffers (and regions therein) of interest.  The
> challenge being in making said specification (somewhat) stable across
> revisions.  I.e., naive linking via absolute line numbers wouldn't be
> sufficient.

What bothers me in this entire discussion is that people are
discussing implementation without first describing the use cases,
requirements, and expectations in enough detail to think about both
the need/importance and possible implementations.

Re: Buffers with buffers (transclusion)

[Vladimir Kazanov]

Hi Eli,

On Thu, 17 Oct 2024 at 18:18, Eli Zaretskii <eliz@gnu.org> wrote:
> What bothers me in this entire discussion is that people are
> discussing implementation without first describing the use cases,
> requirements, and expectations in enough detail to think about both
> the need/importance and possible implementations.

My bad, I should have expanded on the use cases further. So let's come
back to motivating examples.

People have been trying to implement something like we want here for
quite some time. When I wrote that little mode
(https://github.com/vkazanov/chunk-edit), I wasn't aware of all the
previous attempts so it makes sense to cover these in one email.

So let's work from questions. What use could there be for a
buffers-in-buffers function? Or at least an easy way to sync
text/properties between buffers?

Here are some examples of buffer synchronisation as a user-visible feature:

0. chunk-edit.el (https://github.com/vkazanov/chunk-edit) - my humble
prototype that led to this discussion. I know that windows cover this
feature but I also want to point out that people find it inconvenient
when there are more than 3-4 windows to manage.

1. org-transclusion (https://github.com/nobiot/org-transclusion/) is a
popular (1000 stars on Github) org-mode expansion that makes it
possible to mix and match within a single org-mode file chunks of
other org-mode files. It's like chunk-edit.el but focuses on org-mode
only. It provides a live buffer sync feature.

2. Interestingly, subr.el (part of Emacs) has a function called
(text-clone-create) that uses overlays to sync text between regions of
a single buffer. I couldn't find any uses within Emacs, probably
because lack of cross buffer sync limits its usefulness.
org-transclusion contains an expanded version that does support
cross-buffer sync.

Other examples use buffer synchronisation indirectly, as an underlying
mechanism to provide higher level features:

3. In org-mode there is a popular feature called source blocks. Code
within those source blocks needs fontification specific to the
language used in the block. To make fontification possible, org-mode
creates an indirect buffer with the text in the block, enables a
language major mode and copies back relevant properties.

4. polymode (https://github.com/polymode/polymode), a popular
multimode package, famously uses a similar approach by using indirect
buffers and synchronizing both state and *behaviour* back to the host
buffer.

There were multiple discussions in emacs-devel and org-mode mailing
lists around various forms of transclusion that could be helped by
buffers-in-buffers or advanced buffer synchronization mechanisms:

5. A proposal for buffers-in-buffers made by  Dmitrii Korobeinikov
(https://lists.gnu.org/archive/html/emacs-devel/2020-04/msg00141.html).
The author sees this as a way to expand org-mode and computational
notebooks.

6. A brief discussion of something like chunk-edit.el
(https://lists.gnu.org/archive/html/emacs-devel/2018-07/msg00863.html).

7. org-mode transclusion proposal by John Kitchin
(https://lists.gnu.org/archive/html/emacs-orgmode/2016-12/msg00183.html),
which probably lead to creation of the org-transclusion package listed
above.

8. Many other links in a bookmark collection created by Adam Porter
(https://github.com/alphapapa/transclusion-in-emacs). Thanks Adam,
btw, this is an amazing doc!

There are many more examples accumulated over the years which might
benefit from cross-buffer state synchronization support, be it
buffers-in-buffers, or some kind of text/property sync.

Hope this helps with understanding where I come from with this discussion.

For me personally, this gives enough material for inspiration and I am
more than happy to provide a prototype implementation for further
discussion.


-- 
Regards,

Vladimir Kazanov

Re: Buffers with buffers (transclusion)

[Suhail Singh]

Vladimir Kazanov <vekazanov@gmail.com> writes:

>> What bothers me in this entire discussion is that people are
>> discussing implementation without first describing the use cases,
>> requirements, and expectations in enough detail to think about both
>> the need/importance and possible implementations.
>
> 1. org-transclusion (https://github.com/nobiot/org-transclusion/) is a
> popular (1000 stars on Github) org-mode expansion that makes it
> possible to mix and match within a single org-mode file chunks of
> other org-mode files. It's like chunk-edit.el but focuses on org-mode
> only. It provides a live buffer sync feature.

Specifically, I make frequent use of org-transclusion .  At any given
time I may be involved in one or more projects at $work.  I need the
ability to summarize (for a given time period) my contributions towards
a specific project.  Simultaneously, I also need the ability to
summarize my contributions (for a given time period) across all
projects.  By having project-specific Org files where I maintain
project-specific summaries and having a joint Org file that links to the
project-specific summaries and "transcludes" them (i.e., inlines their
specific content) I am able to address both needs.

-- 
Suhail

Re: Buffers with buffers (transclusion)

[James Thomas]

Vladimir Kazanov wrote:

> Eli, James,
>
>> Yes, that's exactly the kind of thoughts I had when reading the
>> description.  Why do we need to display text from different sources in
>> the same buffer, when we already have side-by-side display in adjacent
>> windows?  If the problem is the decoration which set windows apart, it
>> is a minor issue to begin with, and we can make the separating
>> decorations less prominent.
>
> True, having a bunch of stacked windows, or  side by side windows,
> works to an extent, and this is how I do it now. There are downsides
> to this approach.
>
> Firstly, all of the editing always happens in one of the windows, not
> across windows: search/replace, occur, grep. Having things in one
> buffer makes these things as easy as editing a single small file.

There are things like multi-isearch/occur already, no?

> Also, the number of windows that can be comfortably used in this
> manner is limited. 2? 3? 4?  Doing this within a single window is just
> simpler.

Isn't 2 or 3 enough? AFAIU, one at the bottom to phase-in the succeeding
buffer, and the obverse of that. Of course, depending on buffer content.

Regards,
James

Re: Buffers with buffers (transclusion)

[Christopher Howard]

Hi, *chunk-edit* seemed like a very appealing idea to me, so I tried it right away this morning. My use case was editing two chunks of elisp code that were very far apart in a single file.

However, for me the fatal issue is that chunk-edit-mode replaces the active major mode, so that indentation, and presumably other things, are not handled properly. I think any approach to this has to deal with that properly — maintaining the correct major mode in chunk.

It think my particular use case could instead be handled with zones.el. Not sure about the case where you have, e.g., someone wanting to edit a region in a C header file and a region in a C source file, in the same buffer. I suppose that might be useful, e.g., when using search and replace macros and such like.

-- 
Christopher Howard

Re: Buffers with buffers (transclusion)

[Vladimir Kazanov]

Hi Christopher,

Thanks for taking a look at *chunk-edit*! It is an early prototype
which I shared to check if people are interested in this kind of work.

> However, for me the fatal issue is that chunk-edit-mode replaces the active major mode, so that indentation, and presumably other things, are not handled properly. I think any approach to this has to deal with that properly — maintaining the correct major mode in chunk.

One way of developing chunk-edit.el would be to port tricks used in
polymode, which supports having multiple major modes within a single
buffer. For now *chunk-edit* does provide syntax highlighting in the
same way org-mode supports it in src blocks.

> It think my particular use case could instead be handled with zones.el. Not sure about the case where you have, e.g., someone wanting to edit a region in a C header file and a region in a C source file, in the same buffer. I suppose that might be useful, e.g., when using search and replace macros and such like.

Right now I am exploring the best way to bring in chunks of text from
secondary buffers into the host buffer.  This is useful for
*chunk-edit*, and some org-mode ideas, and probably many more things.

-- 
Regards,

Vladimir Kazanov