Request for pointers and advice: displaying several buffers inside a single window

Request for pointers and advice: displaying several buffers inside a single window

[Dmitrii Korobeinikov]

Hi all,

I want to explore the feasibility of displaying two (indirect) buffers
inside a single window, one after another. Each buffer has to look
like it's rendered in its own mode. For the starters, each buffer can
start on its own new line in the window. No interaction w/ text is
required for now.

I have been entertaining the idea since my proposal for lenses [1],
and now that I have some desire to get hacking a little, I want to see
if I can get onto this first ladder step to the goal. What concerns me
now is how to approach the problem. So, I come here to ask for your
advice and comments. You know, "it's dangerous to go alone, take
this!" sort of deal.

I guess I would first have to learn how the renderer functions. Any
good resources on that besides the source code?

Eventually, I would have to explore beyond the renderer and get to
play with the data structures, so I can do the interaction w/ text
possible. But that doesn't concern me all that much until I get to
actually draw some buffers together.

Any man-hour estimates? : )

Best,
DK

[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=35419

Re: Request for pointers and advice: displaying several buffers inside a single window

[Eli Zaretskii]

> From: Dmitrii Korobeinikov <dim1212k@gmail.com>
> Date: Fri, 3 Apr 2020 17:54:07 +0600
> 
> I guess I would first have to learn how the renderer functions. Any
> good resources on that besides the source code?

Basically, none besides the source code.  The large comment at the
beginning of xdisp.c should get you started.  Feel free to ask
questions here, if something is unclear.

Good luck!

Re: Request for pointers and advice: displaying several buffers inside a single window

[Dmitrii Korobeinikov]

That's a quite informative intro, thanks for the reference!

пт, 3 апр. 2020 г. в 19:08, Eli Zaretskii <eliz@gnu.org>:
>
> > From: Dmitrii Korobeinikov <dim1212k@gmail.com>
> > Date: Fri, 3 Apr 2020 17:54:07 +0600
> >
> > I guess I would first have to learn how the renderer functions. Any
> > good resources on that besides the source code?
>
> Basically, none besides the source code.  The large comment at the
> beginning of xdisp.c should get you started.  Feel free to ask
> questions here, if something is unclear.
>
> Good luck!

Re: Request for pointers and advice: displaying several buffers inside a single window

[Adam Porter]

FYI, I've started gathering resources about transclusion in Emacs here:

https://github.com/alphapapa/transclusion-in-emacs

The topic comes up frequently, and there are quite a few resources about
it, but they're scattered, so this tries to collect them in one place.
Patches welcome to add more links!

Re: Request for pointers and advice: displaying several buffers inside a single window

[Dmitrii Korobeinikov]

> https://github.com/alphapapa/transclusion-in-emacs

Never knew about most of the stuff linked there, thanks for organizing
this! Will make sure to send a patch if I stumble upon something not
there already.

PS I don't know how to reply to your message since it hasn't landed in
my inbox (for it not to end up as a seperate thread), so using one of
the previous messages.

пт, 3 апр. 2020 г. в 19:08, Eli Zaretskii <eliz@gnu.org>:
>
> > From: Dmitrii Korobeinikov <dim1212k@gmail.com>
> > Date: Fri, 3 Apr 2020 17:54:07 +0600
> >
> > I guess I would first have to learn how the renderer functions. Any
> > good resources on that besides the source code?
>
> Basically, none besides the source code.  The large comment at the
> beginning of xdisp.c should get you started.  Feel free to ask
> questions here, if something is unclear.
>
> Good luck!

Re: Request for pointers and advice: displaying several buffers inside a single window

[Yuan Fu]

Following mode plus atomic windows seems doable. But it wouldn’t be transparent, AFAIK follow-mode needs to add special handlers for things like isearch and query-replace. OTOH, it would be much more difficult to implement in xdisp.c. I wonder if there is an intermediate layer that’s higher level than redisplay (in lisp, or at least not in redisplay) but still more or less transparent.

Yuan

Re: Request for pointers and advice: displaying several buffers inside a single window

[Ihor Radchenko]

> Following mode plus atomic windows seems doable. But it wouldn’t be
> transparent, AFAIK follow-mode needs to add special handlers for
> things like isearch and query-replace. OTOH, it would be much more
> difficult to implement in xdisp.c. I wonder if there is an
> intermediate layer that’s higher level than redisplay (in lisp, or at
> least not in redisplay) but still more or less transparent. 

What about extending the idea of indirect buffers? In indirect buffers,
the buffer strings are associated with the same memory address storing
the text (if I understand correctly). Similar thing (theoretically) can
be done for individual segments of text. Indeed, there will still be a
question on how the fontification is done, if the overlays should be
shared, and how the key bindings should behave on such segments, but the
basic functionality of automatically sharing text segments between
buffers may be a good framework to start considering more complicated
cases.

Best,
Ihor

Yuan Fu <casouri@gmail.com> writes:

> Following mode plus atomic windows seems doable. But it wouldn’t be transparent, AFAIK follow-mode needs to add special handlers for things like isearch and query-replace. OTOH, it would be much more difficult to implement in xdisp.c. I wonder if there is an intermediate layer that’s higher level than redisplay (in lisp, or at least not in redisplay) but still more or less transparent.
>
> Yuan

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

Re: Request for pointers and advice: displaying several buffers inside a single window

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: adam@alphapapa.net, Eli Zaretskii <eliz@gnu.org>, emacs-devel
>  <emacs-devel@gnu.org>
> Date: Fri, 10 Apr 2020 23:35:12 +0800
> 
> What about extending the idea of indirect buffers? In indirect buffers,
> the buffer strings are associated with the same memory address storing
> the text (if I understand correctly). Similar thing (theoretically) can
> be done for individual segments of text.

I don't think I understand what you mean by "segments of text".
Buffer text is just a long C array of 'char' in Emacs.

Re: Request for pointers and advice: displaying several buffers inside a single window

[Ihor Radchenko]

> I don't think I understand what you mean by "segments of text".
> Buffer text is just a long C array of 'char' in Emacs.

I mean that buffer text in indirect buffer is a pointer to the first
element of the C array containing the text in master buffer (or
something similar). Similarly, the pointer can be created to nth element
of array. Thus, "segment" of text starting from some non-zero position
in the master buffer can be automatically synchronised with the master
buffer. 

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Cc: adam@alphapapa.net, Eli Zaretskii <eliz@gnu.org>, emacs-devel
>>  <emacs-devel@gnu.org>
>> Date: Fri, 10 Apr 2020 23:35:12 +0800
>> 
>> What about extending the idea of indirect buffers? In indirect buffers,
>> the buffer strings are associated with the same memory address storing
>> the text (if I understand correctly). Similar thing (theoretically) can
>> be done for individual segments of text.
>
> I don't think I understand what you mean by "segments of text".
> Buffer text is just a long C array of 'char' in Emacs.

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

Re: Request for pointers and advice: displaying several buffers inside a single window

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: casouri@gmail.com, dim1212k@gmail.com, adam@alphapapa.net,
>  emacs-devel@gnu.org
> Date: Sat, 11 Apr 2020 01:46:59 +0800
> 
> > I don't think I understand what you mean by "segments of text".
> > Buffer text is just a long C array of 'char' in Emacs.
> 
> I mean that buffer text in indirect buffer is a pointer to the first
> element of the C array containing the text in master buffer (or
> something similar). Similarly, the pointer can be created to nth element
> of array. Thus, "segment" of text starting from some non-zero position
> in the master buffer can be automatically synchronised with the master
> buffer. 

So a "segment" is a part of buffer text that starts from some offset,
and then goes on until the end of the buffer text?  IOW, the beginning
is different, but the end is the same?

If so, how is this different from narrowing in indirect buffers that
we have now?

Re: Request for pointers and advice: displaying several buffers inside a single window

[Ihor Radchenko]

> So a "segment" is a part of buffer text that starts from some offset,
> and then goes on until the end of the buffer text?  IOW, the beginning
> is different, but the end is the same?

I would rather say that "segment" is a text between two markers in an
arbitrary buffer. Just like narrowing.

> If so, how is this different from narrowing in indirect buffers that
> we have now?

The difference is that text in a single buffer must not always be coming
from a single buffer (possibly narrowed), but can as well be a list of
"segments". Instead of storing all the buffer text in a single char
array, the buffer text could be represented by a list of "segments". In
a normal buffer, this list will just contain a single element pointing
to the buffer's char array containing its text. However, the list can be
extended with "segments" from other buffers.


Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Cc: casouri@gmail.com, dim1212k@gmail.com, adam@alphapapa.net,
>>  emacs-devel@gnu.org
>> Date: Sat, 11 Apr 2020 01:46:59 +0800
>> 
>> > I don't think I understand what you mean by "segments of text".
>> > Buffer text is just a long C array of 'char' in Emacs.
>> 
>> I mean that buffer text in indirect buffer is a pointer to the first
>> element of the C array containing the text in master buffer (or
>> something similar). Similarly, the pointer can be created to nth element
>> of array. Thus, "segment" of text starting from some non-zero position
>> in the master buffer can be automatically synchronised with the master
>> buffer. 
>
> So a "segment" is a part of buffer text that starts from some offset,
> and then goes on until the end of the buffer text?  IOW, the beginning
> is different, but the end is the same?
>
> If so, how is this different from narrowing in indirect buffers that
> we have now?

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

RE: Request for pointers and advice: displaying several buffers inside a single window

[Drew Adams]

> I would rather say that "segment" is a text between two markers in an
> arbitrary buffer. Just like narrowing.
> 
> > If so, how is this different from narrowing in indirect buffers that
> > we have now?
> 
> The difference is that text in a single buffer must not always be
> coming from a single buffer (possibly narrowed), but can as well be a list of
> "segments". Instead of storing all the buffer text in a single char
> array, the buffer text could be represented by a list of "segments". In
> a normal buffer, this list will just contain a single element pointing
> to the buffer's char array containing its text. However, the list can
> be extended with "segments" from other buffers.

FWIW, this is what library zones.el offers.  The zones ("segments") in a given set of zones can be for the same buffer or different buffers.

But if your question/request is about displaying text from different buffers in the same Emacs window then zones.el won't help you.

See also library narrow-indirect.el.

https://www.emacswiki.org/emacs/download/zones.el

https://www.emacswiki.org/emacs/download/narrow-indirect.el

RE: Request for pointers and advice: displaying several buffers inside a single window

[Ihor Radchenko]

> FWIW, this is what library zones.el offers.  The zones ("segments") in a given set of zones can be for the same buffer or different buffers.

Thanks for the reference. I tried zones once, but abandoned exactly
because I was looking for showing all zones at once (not cycling through
them). 

> But if your question/request is about displaying text from different buffers in the same Emacs window then zones.el won't help you.

Yep. Displaying and editing (with synchronised changes in those
buffers).


Drew Adams <drew.adams@oracle.com> writes:

>> I would rather say that "segment" is a text between two markers in an
>> arbitrary buffer. Just like narrowing.
>> 
>> > If so, how is this different from narrowing in indirect buffers that
>> > we have now?
>> 
>> The difference is that text in a single buffer must not always be
>> coming from a single buffer (possibly narrowed), but can as well be a list of
>> "segments". Instead of storing all the buffer text in a single char
>> array, the buffer text could be represented by a list of "segments". In
>> a normal buffer, this list will just contain a single element pointing
>> to the buffer's char array containing its text. However, the list can
>> be extended with "segments" from other buffers.
>
> FWIW, this is what library zones.el offers.  The zones ("segments") in a given set of zones can be for the same buffer or different buffers.
>
> But if your question/request is about displaying text from different buffers in the same Emacs window then zones.el won't help you.
>
> See also library narrow-indirect.el.
>
> https://www.emacswiki.org/emacs/download/zones.el
>
> https://www.emacswiki.org/emacs/download/narrow-indirect.el

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

RE: Request for pointers and advice: displaying several buffers inside a single window

[Drew Adams]

> I tried zones once, but abandoned exactly
> because I was looking for showing all zones at once (not cycling
> through them).

To be clear, there's no need to cycle among zones.
No need to see only one at a time. You can see all
of them all of the time, as well as see the other,
non-zone text, of any and all buffers.

What zones.el does not offer is showing the text
of more than one buffer in the same Emacs window.

> > But if your question/request is about displaying text from different
> buffers in the same Emacs window then zones.el won't help you.
> 
> Yep. Displaying and editing (with synchronised changes in those
> buffers).

See above.  You can display and edit all zones at
the same time.  And there's no synchronization
needed.  What you can't do with zones is show and
edit the text of more than one buffer in the same
Emacs window.

I say all of this only in case it helps somehow.
My intention is not to have you use zones.el.

Maybe consider this feedback as just letting you
know that I, at least, don't quite understand
what you're trying to do (or why).

Emacs doesn't let you use the same window for
multiple buffers, as far as I know.

You can finagle ways to show text from multiple
buffers in the same window, e.g. by copying it.
And if you do that, and you then want to edit
the copies, then, yes, you'll need to then sync
up the original buffers with your edits.

Maybe there's a library that does that for you.

There are libraries that let you put different
parts of the same buffer in different major
modes (in effect).  But that too is different
from having multiple buffers in the same window.

I wonder what your reason is for wanting that?
That "why" might help explain your request.

RE: Request for pointers and advice: displaying several buffers inside a single window

[Ihor Radchenko]

> To be clear, there's no need to cycle among zones.
> No need to see only one at a time. You can see all
> of them all of the time, as well as see the other,
> non-zone text, of any and all buffers.
>
> What zones.el does not offer is showing the text
> of more than one buffer in the same Emacs window.

I was not able to achieve this. Do you mean that zones located more than
one screen away in the same buffer can be shown all together resulting
in a narrowing with multiple zones shown one after other without text
between them?

> Maybe consider this feedback as just letting you
> know that I, at least, don't quite understand
> what you're trying to do (or why).
>
> Emacs doesn't let you use the same window for
> multiple buffers, as far as I know.

I am trying to suggest something for displaying text from multiple
buffers in a single window. My idea is to modify Emacs buffer internals
to achieve this.

> Emacs doesn't let you use the same window for
> multiple buffers, as far as I know.
>
> You can finagle ways to show text from multiple
> buffers in the same window, e.g. by copying it.
> And if you do that, and you then want to edit
> the copies, then, yes, you'll need to then sync
> up the original buffers with your edits.
>
> Maybe there's a library that does that for you.

I am not aware of such library, except the one I am working on. However,
any lisp level implementation is likely to slow Emacs more than if it is
done on C level.

> I wonder what your reason is for wanting that?
> That "why" might help explain your request.

I think the reasons were discussed in
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=35419 and popped up
several times in internet:

- https://github.com/alphapapa/transclusion-in-emacs/
- https://lists.gnu.org/archive/html/emacs-devel/2018-07/msg00863.html
- https://www.reddit.com/r/orgmode/comments/fimdf7/nesting_many_org_source_files_into_the_same_window/

For me, the reasons are mostly related to org mode. For example, it
would be cool to have the same org heading in multiple places (and be
able to edit the heading from any of those places).


Drew Adams <drew.adams@oracle.com> writes:

>> I tried zones once, but abandoned exactly
>> because I was looking for showing all zones at once (not cycling
>> through them).
>
> To be clear, there's no need to cycle among zones.
> No need to see only one at a time. You can see all
> of them all of the time, as well as see the other,
> non-zone text, of any and all buffers.
>
> What zones.el does not offer is showing the text
> of more than one buffer in the same Emacs window.
>
>> > But if your question/request is about displaying text from different
>> buffers in the same Emacs window then zones.el won't help you.
>> 
>> Yep. Displaying and editing (with synchronised changes in those
>> buffers).
>
> See above.  You can display and edit all zones at
> the same time.  And there's no synchronization
> needed.  What you can't do with zones is show and
> edit the text of more than one buffer in the same
> Emacs window.
>
> I say all of this only in case it helps somehow.
> My intention is not to have you use zones.el.
>
> Maybe consider this feedback as just letting you
> know that I, at least, don't quite understand
> what you're trying to do (or why).
>
> Emacs doesn't let you use the same window for
> multiple buffers, as far as I know.
>
> You can finagle ways to show text from multiple
> buffers in the same window, e.g. by copying it.
> And if you do that, and you then want to edit
> the copies, then, yes, you'll need to then sync
> up the original buffers with your edits.
>
> Maybe there's a library that does that for you.
>
> There are libraries that let you put different
> parts of the same buffer in different major
> modes (in effect).  But that too is different
> from having multiple buffers in the same window.
>
> I wonder what your reason is for wanting that?
> That "why" might help explain your request.

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

RE: Request for pointers and advice: displaying several buffers inside a single window

[Drew Adams]

> > To be clear, there's no need to cycle among zones.
> > No need to see only one at a time. You can see all
> > of them all of the time, as well as see the other,
> > non-zone text, of any and all buffers.
> >
> > What zones.el does not offer is showing the text
> > of more than one buffer in the same Emacs window.
> 
> I was not able to achieve this. Do you mean that zones
> located more than one screen away in the same buffer
> can be shown all together resulting in a narrowing with
> multiple zones shown one after other without text
> between them?

No, not exactly; not a multi-narrowing.

I meant only that you need not access a zone or
make it visible only by narrowing to it.  Without
narrowing, zones are still defined, and you can use
them and act on them in an infinite number of ways.
Zones are just defined areas of a buffer.

But as for your question about seeing only some (or
all) zones, and hiding the text between them: That's
indeed possible, but it's not about narrowing.

It's instead about making a set of zones (or its
complement) _invisible_.  For this you also need
library `isearch-prop.el'.  If you load that library
then you can use these keys on prefix key C-x n M-=:

v - isearchp-toggle-anti-zones-invisible
V - isearchp-toggle-zones-invisible
~ - isearchp-toggle-complementing-domain
d - isearchp-toggle-dimming-outside-search-area

The first of these toggles hiding (making invisible)
the complement of (the union of) the current set of
zones, that is, the anti-zones.

With a prefix arg it also toggles visibility of the
zones themselves the other way (e.g. makes them
visible when it makes the anti-zones invisible, and
vice versa).

Invisibility, here, is the usual Emacs invisibility
of text.

So for example, if you have a bunch of zones in a
given buffer, and you use `C-x n M-= v', then all
of the text outside those zones (the anti-zones)
is made invisible (disappears).  You see the zones
right next to each other, with no intervening text.  Repeating `C-x n M-= v' shows the anti-zones again.

> > Maybe consider this feedback as just letting you
> > know that I, at least, don't quite understand
> > what you're trying to do (or why).
> >
> > Emacs doesn't let you use the same window for
> > multiple buffers, as far as I know.
> 
> I am trying to suggest something for displaying
> text from multiple buffers in a single window.
> My idea is to modify Emacs buffer internals
> to achieve this.

I understand that.

> > Emacs doesn't let you use the same window for
> > multiple buffers, as far as I know.
> >
> > You can finagle ways to show text from multiple
> > buffers in the same window, e.g. by copying it.
> > And if you do that, and you then want to edit
> > the copies, then, yes, you'll need to then sync
> > up the original buffers with your edits.
> 
> > I wonder what your reason is for wanting that?
> > That "why" might help explain your request.
> 
> I think the reasons were discussed in ... and
> popped up several times in internet...

Yes, I've seen those.

FWIW, I agree that being able to do arbitrary
editing (e.g. search-&-replace) in such a
context would be useful.

That we're talking about a single window here
in effect means we're talking about having a
window that shows a buffer that is like an
indirect buffer that refers to parts of
multiple buffers.  One way to think of it
could be as an extension of the notion of
indirect buffer.

zones.el doesn't help with this.  It does let
you do such things for zones in the _same_
buffer.  And it does let you work on sets of
zones across multiple buffers.  But it doesn't
let you work on the latter in the same window
(i.e., the same ~indirect buffer for multiple
buffers).

> For me, the reasons are mostly related to org
> mode. For example, it would be cool to have
> the same org heading in multiple places (and be
> able to edit the heading from any of those places).

I can see that use case.  But I'd encourage
people to think beyond Org mode use cases for
the kind of thing being discussed.  Being able
to have, in effect, an indirect buffer that
refers to multiple buffers is _much_ more
general than any Org mode uses.

(I'm saying "indirect buffer" here, but I
know that indirect buffers currently are
limited.  They are in some ways too tightly
related to their base buffers.)

In a way, Org mode tries to give you some
similar behavior, by delimiting buffer areas
using plain-text tags (similar to what XML
tags do).

The approach taken by zones.el is better in
this regard, I think.  It defines zones only
by buffer and positions (which can be markers
or Lisp-readable markers).  A zone is like an
overlay, but it has an identifier, it can be
Lisp-readable and persistent, and it can be
buffer-independent (used in different buffers).

Anyway, good luck with your project.  I hope
it will ultimately be general enough to help
with _all_ of the various uses people have
envisioned for acting on areas of different
buffers in the same Emacs window / indirect
buffer.  I wouldn't like to see something that
is limited to something like Org mode, or is
limited to use with multiple major modes.  I'd
like to see something very general and flexible.

Re: Request for pointers and advice: displaying several buffers inside a single window

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   If you load that library
  > then you can use these keys on prefix key C-x n M-=:

  > v - isearchp-toggle-anti-zones-invisible
  > V - isearchp-toggle-zones-invisible
  > ~ - isearchp-toggle-complementing-domain
  > d - isearchp-toggle-dimming-outside-search-area

Can anyone come up with a simpler interface for this?
One that reduces the amount that a user needs to memorize
before using the feature?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

RE: Request for pointers and advice: displaying several buffers inside a single window

[Drew Adams]

>   > If you load that library then you can use
>   > these keys on prefix key C-x n M-=:
>   >
>   > v - isearchp-toggle-anti-zones-invisible
>   > V - isearchp-toggle-zones-invisible
>   > ~ - isearchp-toggle-complementing-domain
>   > d - isearchp-toggle-dimming-outside-search-area
> 
> Can anyone come up with a simpler interface for this?
> One that reduces the amount that a user needs to memorize
> before using the feature?

You don't need to memorize anything.  And certainly
not in order to be able to use the feature.  The
predefined keys are just for convenience.

And you can bind any of the commands to whatever
keys you like.

By default, the zone commands are on prefix key
`C-x n', which Emacs uses for narrowing commands
(Emacs binds it to keymap `narrow-map').

(If you use isearch-prop.el or highlight.el then
zones.el adds additional zone commands to the same
keymap.)

I chose that keymap and prefix key for zones mainly
because zones.el optionally advises the standard
narrowing commands (`narrow-region' etc.), so they 
automatically add a narrowing (restriction) as a
zone.  (You can then cycle among restrictions etc.)

But you can use any other keymap or prefix key.

I decided to put all of the toggle commands for use
with zones on keymap `zz-toggles-map'.  And I put
that map on prefix key `M-=' in `narrow-map'.  But
you can use a different prefix key for that map, or
ignore it.

If you do want to use the predefined bindings, and
you don't remember some, using either `C-x n C-h' or
`C-h M-k narrow-map' (`describe-keymap') tells you:

narrow-map
----------

Keymap for narrowing commands.

key             binding
---             -------

C-d             zz-delete-zone
C-r             isearchp-zones-backward
C-s             isearchp-zones-forward
ESC             Prefix Command
#               zz-select-zone-by-id-and-text
A               zz-add-zone-and-unite
C               zz-clone-and-unite-zones
D               isearchp-remove-dimming
H               hlt-highlight-regions-in-buffers
L               zz-set-zones-from-highlighting
P               isearchp-put-prop-on-zones
R               zz-set-zones-matching-regexp
a               zz-add-zone
c               zz-clone-zones
d               narrow-to-defun
h               hlt-highlight-regions
l               zz-add-zones-from-highlighting
n               narrow-to-region
p               narrow-to-page
r               zz-add-zones-matching-regexp
s               zz-select-zone-repeat
u               zz-unite-zones
v               zz-set-izones-var
w               widen
x               zz-narrow-repeat

C-M-r           isearchp-zones-backward-regexp
C-M-s           isearchp-zones-forward-regexp
M-%             zz-query-replace-zones
M-=             zz-toggles-map
C-M-%           zz-query-replace-regexp-zones

M-= V           isearchp-toggle-zones-invisible
M-= d           isearchp-toggle-dimming-outside-search-area
M-= v           isearchp-toggle-anti-zones-invisible
M-= ~           isearchp-toggle-complementing-domain

RE: Request for pointers and advice: displaying several buffers inside a single window

[Drew Adams]

I meant to include links to these Emacs-Wiki
pages, which describe the keys/commands a bit
and the relation to narrowing.

https://www.emacswiki.org/emacs/Zones

https://www.emacswiki.org/emacs/MultipleNarrowings

Re: Request for pointers and advice: displaying several buffers inside a single window

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > You don't need to memorize anything.  And certainly
  > not in order to be able to use the feature.  The
  > predefined keys are just for convenience.

  > And you can bind any of the commands to whatever
  > keys you like.

I think we are miscommunicating.  All Emacs key bindings are "just for
convenience" and each user can rebind them.  Notwithstanding that, it
is important for default bindings to be convenient.  C-x n M-= v is
not going to be easy to remember.  If a command is worth giving a key
binding to, can someone propose a more convenient default binding?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

RE: Request for pointers and advice: displaying several buffers inside a single window

[Drew Adams]

>> You don't need to memorize anything.  And certainly
>> not in order to be able to use the feature.  The
>> predefined keys are just for convenience.
> 
>> And you can bind any of the commands to whatever
>> keys you like.
> 
> I think we are miscommunicating.  All Emacs key bindings are "just for
> convenience" and each user can rebind them.  Notwithstanding that, it
> is important for default bindings to be convenient.  C-x n M-= v is
> not going to be easy to remember.  If a command is worth giving a key
> binding to, can someone propose a more convenient default binding?

I'm the one who decided that the commands are worth
giving key bindings to.  I believe it's convenient
enough to put all zone keys on `narrow-map', and I
believe it's convenient enough to put all toggle
commands for zones on their own keymap, and to put
that on `narrow-map'.

Putting the toggle keys on their own map makes it
easy for someone to move them, together, to another
prefix key.

Putting all zone keys on an _existing_ prefix key
means saving another prefix key for some other use.
`narrow-map' makes sense for that, because it's
about multiple different buffer areas: ~zones.

Where we may be miscommunicating is perhaps that
it's up to me to decide which keys I think best to
predefine for zones.el and isearch-prop.el.

I'm quite open to user feedback, including about
predefined key bindings.  But I'm not convinced by
the argument that `C-x n M-=' shouldn't be used as
a default prefix key because it's not convenient.

My priorities here are not just for quick keys or
individually memorable keys.  I expect users to
pick their own keys for that, if they like, based
on their use.

My priorities for this are instead:

1. Group all zone keys on a prefix key.
2. Don't waste a new prefix key - use existing key.
2. Use `C-x n', because:
   (a) it's relevant - logical association between
       zones and buffer restrictions
   (b) it doesn't already have many keys on it
3. Group the toggle keys on their own prefix key
   (which is on `C-x n').

#2 and #3 do help memory, but for key _groups_,
not for individual keys.

For individual toggle keys there are also mnemonics:

`v', `V' for visibility toggles
`~' ("not") for complementing the domain
`d' for dimming

Similarly for the other zone keys, these mnemonics:

`#' for selecting zones by ID number
`a', `A' for adding zones
`c', `C' for cloning zones
`C-d' for deleting zones from the current zones set
`D' for deleting text-dimming
`h', `H' for highlighting zones
`l', `L' for adding/setting zones from (high)lighting
`P' for property-putting on zones
`r', `R' for adding/setting zones from regexp matches
`s' for selecting zones as the region
`u' for uniting (coalescing) zones
`v' for changing the current zones variable

And these, which follow the usual search keys:

`C-s', `C-M-s', `C-r', `C-M-r' for search
`M-%', `C-M-%' for query-replace

The one key that's not really mnemonic, but which
gets used a lot, is `x', which cycles among
narrowings (`C-x n x x x...).

I would have picked `n' for that, which _is_
mnemonic, but that's already `narrow-to-region' (not
especially mnemonic within the set of narrowing
keys, but it was perhaps the first such).

I picked `D' for dimming because `d' was already
`narrow-to-defun'.

I picked `P' for property-putting because `p' was
already `narrow-to-page'.

Re: Request for pointers and advice: displaying several buffers inside a single window

[Juri Linkov]

>   > v - isearchp-toggle-anti-zones-invisible
>   > V - isearchp-toggle-zones-invisible
>   > ~ - isearchp-toggle-complementing-domain
>   > d - isearchp-toggle-dimming-outside-search-area
>
> Can anyone come up with a simpler interface for this?
> One that reduces the amount that a user needs to memorize
> before using the feature?

Isearch keybindings could be on the M-s prefix key.  query-replace already
supports zones (as a subset of rectangular regions), so isearch could as well.

RE: Request for pointers and advice: displaying several buffers inside a single window

[Drew Adams]

> >   > v - isearchp-toggle-anti-zones-invisible
> >   > V - isearchp-toggle-zones-invisible
> >   > ~ - isearchp-toggle-complementing-domain
> >   > d - isearchp-toggle-dimming-outside-search-area
> >
> > Can anyone come up with a simpler interface for this?
> > One that reduces the amount that a user needs to memorize
> > before using the feature?
> 
> Isearch keybindings could be on the M-s prefix key.

Kind of you to worry about this.  But the library
that defines those toggle commands, isearch-prop.el,
already binds the two non-zone ones to keys in
`isearch-mode-map':

(define-key isearch-mode-map (kbd "C-M-~")
            'isearchp-toggle-complementing-domain)
(define-key isearch-mode-map (kbd "C-M-S-d")
            'isearchp-toggle-dimming-outside-search-area)

Those two commands are not only for zones.  They apply
also to search, so they deserve `isearch-mode-map' keys.

I gave them bindings in the zones toggle keymap because
they're also useful for zones, not just for searching.
Both sets of bindings are useful.

(Dimming the areas outside zones and dimming the areas
not being searched have the same behavior.  I defined
it first for searching, which is why its toggle command
still has the name `...outside-search-area'.  Maybe I
should rename it.)

Yes, I'm the one who decided which of the zones.el and
isearch-prop.el commands deserve keys, and in which maps.

RE: Request for pointers and advice: displaying several buffers inside a single window

[Ihor Radchenko]

> But as for your question about seeing only some (or
> all) zones, and hiding the text between them: That's
> indeed possible, but it's not about narrowing.

Got it.

> It's instead about making a set of zones (or its
> complement) _invisible_.  For this you also need
> library `isearch-prop.el'.  If you load that library
> then you can use these keys on prefix key C-x n M-=:

I missed it when reading commentary. Probably because I was mostly
looking at key bindings part. I guess that it would be helpful to add
these key bindings to **Keys** section of the commentary, similarly to
highlight.el-related bindings.

> I can see that use case.  But I'd encourage
> people to think beyond Org mode use cases for
> the kind of thing being discussed.  Being able
> to have, in effect, an indirect buffer that
> refers to multiple buffers is _much_ more
> general than any Org mode uses.

I agree that the discussed concept can be used far beyond org-mode
scope. (Otherwise, there would be little reason to discuss adding it to
Emacs core.) I just shared my personal use case (in addition to the
provided links).

> Anyway, good luck with your project.  I hope
> it will ultimately be general enough to help
> with _all_ of the various uses people have
> envisioned for acting on areas of different
> buffers in the same Emacs window / indirect
> buffer.  I wouldn't like to see something that
> is limited to something like Org mode, or is
> limited to use with multiple major modes.  I'd
> like to see something very general and flexible.

FYI, my pre-alpha implementation of synchronised text is in github
(https://github.com/yantar92/mirror-text). If you (or anyone else in the
thread) is interested, feedback is welcome.


Drew Adams <drew.adams@oracle.com> writes:

>> > To be clear, there's no need to cycle among zones.
>> > No need to see only one at a time. You can see all
>> > of them all of the time, as well as see the other,
>> > non-zone text, of any and all buffers.
>> >
>> > What zones.el does not offer is showing the text
>> > of more than one buffer in the same Emacs window.
>> 
>> I was not able to achieve this. Do you mean that zones
>> located more than one screen away in the same buffer
>> can be shown all together resulting in a narrowing with
>> multiple zones shown one after other without text
>> between them?
>
> No, not exactly; not a multi-narrowing.
>
> I meant only that you need not access a zone or
> make it visible only by narrowing to it.  Without
> narrowing, zones are still defined, and you can use
> them and act on them in an infinite number of ways.
> Zones are just defined areas of a buffer.
>
> But as for your question about seeing only some (or
> all) zones, and hiding the text between them: That's
> indeed possible, but it's not about narrowing.
>
> It's instead about making a set of zones (or its
> complement) _invisible_.  For this you also need
> library `isearch-prop.el'.  If you load that library
> then you can use these keys on prefix key C-x n M-=:
>
> v - isearchp-toggle-anti-zones-invisible
> V - isearchp-toggle-zones-invisible
> ~ - isearchp-toggle-complementing-domain
> d - isearchp-toggle-dimming-outside-search-area
>
> The first of these toggles hiding (making invisible)
> the complement of (the union of) the current set of
> zones, that is, the anti-zones.
>
> With a prefix arg it also toggles visibility of the
> zones themselves the other way (e.g. makes them
> visible when it makes the anti-zones invisible, and
> vice versa).
>
> Invisibility, here, is the usual Emacs invisibility
> of text.
>
> So for example, if you have a bunch of zones in a
> given buffer, and you use `C-x n M-= v', then all
> of the text outside those zones (the anti-zones)
> is made invisible (disappears).  You see the zones
> right next to each other, with no intervening text.  Repeating `C-x n M-= v' shows the anti-zones again.
>
>> > Maybe consider this feedback as just letting you
>> > know that I, at least, don't quite understand
>> > what you're trying to do (or why).
>> >
>> > Emacs doesn't let you use the same window for
>> > multiple buffers, as far as I know.
>> 
>> I am trying to suggest something for displaying
>> text from multiple buffers in a single window.
>> My idea is to modify Emacs buffer internals
>> to achieve this.
>
> I understand that.
>
>> > Emacs doesn't let you use the same window for
>> > multiple buffers, as far as I know.
>> >
>> > You can finagle ways to show text from multiple
>> > buffers in the same window, e.g. by copying it.
>> > And if you do that, and you then want to edit
>> > the copies, then, yes, you'll need to then sync
>> > up the original buffers with your edits.
>> 
>> > I wonder what your reason is for wanting that?
>> > That "why" might help explain your request.
>> 
>> I think the reasons were discussed in ... and
>> popped up several times in internet...
>
> Yes, I've seen those.
>
> FWIW, I agree that being able to do arbitrary
> editing (e.g. search-&-replace) in such a
> context would be useful.
>
> That we're talking about a single window here
> in effect means we're talking about having a
> window that shows a buffer that is like an
> indirect buffer that refers to parts of
> multiple buffers.  One way to think of it
> could be as an extension of the notion of
> indirect buffer.
>
> zones.el doesn't help with this.  It does let
> you do such things for zones in the _same_
> buffer.  And it does let you work on sets of
> zones across multiple buffers.  But it doesn't
> let you work on the latter in the same window
> (i.e., the same ~indirect buffer for multiple
> buffers).
>
>> For me, the reasons are mostly related to org
>> mode. For example, it would be cool to have
>> the same org heading in multiple places (and be
>> able to edit the heading from any of those places).
>
> I can see that use case.  But I'd encourage
> people to think beyond Org mode use cases for
> the kind of thing being discussed.  Being able
> to have, in effect, an indirect buffer that
> refers to multiple buffers is _much_ more
> general than any Org mode uses.
>
> (I'm saying "indirect buffer" here, but I
> know that indirect buffers currently are
> limited.  They are in some ways too tightly
> related to their base buffers.)
>
> In a way, Org mode tries to give you some
> similar behavior, by delimiting buffer areas
> using plain-text tags (similar to what XML
> tags do).
>
> The approach taken by zones.el is better in
> this regard, I think.  It defines zones only
> by buffer and positions (which can be markers
> or Lisp-readable markers).  A zone is like an
> overlay, but it has an identifier, it can be
> Lisp-readable and persistent, and it can be
> buffer-independent (used in different buffers).
>
> Anyway, good luck with your project.  I hope
> it will ultimately be general enough to help
> with _all_ of the various uses people have
> envisioned for acting on areas of different
> buffers in the same Emacs window / indirect
> buffer.  I wouldn't like to see something that
> is limited to something like Org mode, or is
> limited to use with multiple major modes.  I'd
> like to see something very general and flexible.

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

RE: Request for pointers and advice: displaying several buffers inside a single window

[Drew Adams]

> > It's instead about making a set of zones (or its
> > complement) _invisible_.  For this you also need
> > library `isearch-prop.el'.  If you load that library
> > then you can use these keys on prefix key C-x n M-=:
> 
> I missed it when reading commentary. Probably because I was mostly
> looking at key bindings part. I guess that it would be helpful to add
> these key bindings to **Keys** section of the commentary, similarly to
> highlight.el-related bindings.

Thanks for that reminder.  I'll do that.  Some other
bindings were also missing from that list.

> FYI, my pre-alpha implementation of synchronised text is in github
> (https://urldefense.com/v3/__https://github.com/yantar92/mirror-
> text__;!!GqivPVa7Brio!NAEaJYE8jQeeMRKcdlSgwbLH9MHO7fBjwT6Ut37aw2uL39Bao
> pDCQxxPeOErp4Sy$ ). If you (or anyone else in the
> thread) is interested, feedback is welcome.

Thanks. I'll take a look.

Re: Request for pointers and advice: displaying several buffers inside a single window

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: casouri@gmail.com, dim1212k@gmail.com, adam@alphapapa.net,
>  emacs-devel@gnu.org
> Date: Sat, 11 Apr 2020 02:37:13 +0800
> 
> > So a "segment" is a part of buffer text that starts from some offset,
> > and then goes on until the end of the buffer text?  IOW, the beginning
> > is different, but the end is the same?
> 
> I would rather say that "segment" is a text between two markers in an
> arbitrary buffer.

So there's one more parameter, in addition to the offset?

> > If so, how is this different from narrowing in indirect buffers that
> > we have now?
> 
> The difference is that text in a single buffer must not always be coming
> from a single buffer (possibly narrowed), but can as well be a list of
> "segments". Instead of storing all the buffer text in a single char
> array, the buffer text could be represented by a list of "segments". In
> a normal buffer, this list will just contain a single element pointing
> to the buffer's char array containing its text. However, the list can be
> extended with "segments" from other buffers.

Sounds like a thorough surgery of all the low-level functions that
handle insertion, deletion, replacement etc.

And what do you suggest to do with the gap?

Re: Request for pointers and advice: displaying several buffers inside a single window

[Ihor Radchenko]

> So there's one more parameter, in addition to the offset?

Sure, there should be at least two parameters to define segment. I am
thinking aloud here. Started with rough idea. I think Dmitrii just
described similar approach and also did some source code lookup. 

> Sounds like a thorough surgery of all the low-level functions that
> handle insertion, deletion, replacement etc.

Yeah. Also, regions, markers, and point motion.

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Cc: casouri@gmail.com, dim1212k@gmail.com, adam@alphapapa.net,
>>  emacs-devel@gnu.org
>> Date: Sat, 11 Apr 2020 02:37:13 +0800
>> 
>> > So a "segment" is a part of buffer text that starts from some offset,
>> > and then goes on until the end of the buffer text?  IOW, the beginning
>> > is different, but the end is the same?
>> 
>> I would rather say that "segment" is a text between two markers in an
>> arbitrary buffer.
>
> So there's one more parameter, in addition to the offset?
>
>> > If so, how is this different from narrowing in indirect buffers that
>> > we have now?
>> 
>> The difference is that text in a single buffer must not always be coming
>> from a single buffer (possibly narrowed), but can as well be a list of
>> "segments". Instead of storing all the buffer text in a single char
>> array, the buffer text could be represented by a list of "segments". In
>> a normal buffer, this list will just contain a single element pointing
>> to the buffer's char array containing its text. However, the list can be
>> extended with "segments" from other buffers.
>
> Sounds like a thorough surgery of all the low-level functions that
> handle insertion, deletion, replacement etc.
>
> And what do you suggest to do with the gap?

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

Re: Request for pointers and advice: displaying several buffers inside a single window

[Ihor Radchenko]

> And what do you suggest to do with the gap?

This is where the buffer text is stored internally right? I think a
"segment" may as well refer to a substring from the gap. In the simplest
case the list of "segments" would contain a single element referring to
the whole gap. Then, if one adds another "segment", the list will hold 3
elements: (<segment from beginning of the gap to pos> <added segment>
<segment referring to the remaining part of the gap>).

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Cc: casouri@gmail.com, dim1212k@gmail.com, adam@alphapapa.net,
>>  emacs-devel@gnu.org
>> Date: Sat, 11 Apr 2020 02:37:13 +0800
>> 
>> > So a "segment" is a part of buffer text that starts from some offset,
>> > and then goes on until the end of the buffer text?  IOW, the beginning
>> > is different, but the end is the same?
>> 
>> I would rather say that "segment" is a text between two markers in an
>> arbitrary buffer.
>
> So there's one more parameter, in addition to the offset?
>
>> > If so, how is this different from narrowing in indirect buffers that
>> > we have now?
>> 
>> The difference is that text in a single buffer must not always be coming
>> from a single buffer (possibly narrowed), but can as well be a list of
>> "segments". Instead of storing all the buffer text in a single char
>> array, the buffer text could be represented by a list of "segments". In
>> a normal buffer, this list will just contain a single element pointing
>> to the buffer's char array containing its text. However, the list can be
>> extended with "segments" from other buffers.
>
> Sounds like a thorough surgery of all the low-level functions that
> handle insertion, deletion, replacement etc.
>
> And what do you suggest to do with the gap?

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

Re: Request for pointers and advice: displaying several buffers inside a single window

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: casouri@gmail.com, dim1212k@gmail.com, adam@alphapapa.net,
>  emacs-devel@gnu.org
> Date: Sat, 11 Apr 2020 03:34:30 +0800
> 
> > And what do you suggest to do with the gap?
> 
> This is where the buffer text is stored internally right?

That's not the definition I'd use.  The gap is a way of making text
insertion less expensive.  If you insert characters one by one, each
insertion needs to move the characters after the insertion point,
which is expensive.  Having the gap allows us to perform this movement
only after relatively large amounts of text were inserted; for smaller
insertions we just make the gap smaller.

> I think a
> "segment" may as well refer to a substring from the gap. In the simplest
> case the list of "segments" would contain a single element referring to
> the whole gap. Then, if one adds another "segment", the list will hold 3
> elements: (<segment from beginning of the gap to pos> <added segment>
> <segment referring to the remaining part of the gap>).

My question was how you handle insertion in a segment?  Move the gap
to the segment, or have a separate gap for each segment?

Re: Request for pointers and advice: displaying several buffers inside a single window

[Ihor Radchenko]

> That's not the definition I'd use.  The gap is a way of making text
> insertion less expensive.  If you insert characters one by one, each
> insertion needs to move the characters after the insertion point,
> which is expensive.  Having the gap allows us to perform this movement
> only after relatively large amounts of text were inserted; for smaller
> insertions we just make the gap smaller.

Do you mean that the gap is specially allocated memory space associated
with buffer point where you can insert characters without a need to
shift the tail of the char array containing buffer string?

> My question was how you handle insertion in a segment?  Move the gap
> to the segment, or have a separate gap for each segment?

A segment could mean two thing as I imagine: (1) pointer to original
char array associated with current buffer. you can think about it as
about narrowed buffer for current buffer implementation; (2) a pair of
markers in different buffer. The second type of segment may refer to a
char array in different buffer (just like case (1), but the char array
from different buffer). Alternatively, it may refer to several other
segments (if other buffer contains several segments between the
markers). The last case can be reduced to a list of char arrays in
different buffers.

If we think about insertion, the point may be within a single segment or
just between two segments. If it is within a single segment, we can
identify (maybe recursively) what is the basic char array corresponding
to the point. The gap will be that gap associated with that char array.
In the other case, when the point is in between two segments, it narrows
down to point position between two char arrays. One of this arrays (and
the associated gap) will be selected depending on some criteria (maybe,
something similar to what we have for overlays/sticky text properties).

I hope that my explanation is clear enough.


Eli Zaretskii <eliz@gnu.org> writes:

>> From: Ihor Radchenko <yantar92@gmail.com>
>> Cc: casouri@gmail.com, dim1212k@gmail.com, adam@alphapapa.net,
>>  emacs-devel@gnu.org
>> Date: Sat, 11 Apr 2020 03:34:30 +0800
>> 
>> > And what do you suggest to do with the gap?
>> 
>> This is where the buffer text is stored internally right?
>
> That's not the definition I'd use.  The gap is a way of making text
> insertion less expensive.  If you insert characters one by one, each
> insertion needs to move the characters after the insertion point,
> which is expensive.  Having the gap allows us to perform this movement
> only after relatively large amounts of text were inserted; for smaller
> insertions we just make the gap smaller.
>
>> I think a
>> "segment" may as well refer to a substring from the gap. In the simplest
>> case the list of "segments" would contain a single element referring to
>> the whole gap. Then, if one adds another "segment", the list will hold 3
>> elements: (<segment from beginning of the gap to pos> <added segment>
>> <segment referring to the remaining part of the gap>).
>
> My question was how you handle insertion in a segment?  Move the gap
> to the segment, or have a separate gap for each segment?

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

Re: Request for pointers and advice: displaying several buffers inside a single window

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: casouri@gmail.com, dim1212k@gmail.com, adam@alphapapa.net,
>  emacs-devel@gnu.org
> Date: Sat, 11 Apr 2020 16:35:58 +0800
> 
> > That's not the definition I'd use.  The gap is a way of making text
> > insertion less expensive.  If you insert characters one by one, each
> > insertion needs to move the characters after the insertion point,
> > which is expensive.  Having the gap allows us to perform this movement
> > only after relatively large amounts of text were inserted; for smaller
> > insertions we just make the gap smaller.
> 
> Do you mean that the gap is specially allocated memory space associated
> with buffer point where you can insert characters without a need to
> shift the tail of the char array containing buffer string?

Yes.  Except that point doesn't need to be at that place, because we
can insert at position that is not necessarily that of point; we move
the gap to the insertion place when we need.

Re: Request for pointers and advice: displaying several buffers inside a single window

[Dmitrii Korobeinikov]

> Following mode plus atomic windows seems doable. But it wouldn’t be
> transparent, AFAIK follow-mode needs to add special handlers for
> things like isearch and query-replace. OTOH, it would be much more
> difficult to implement in xdisp.c. I wonder if there is an
> intermediate layer that’s higher level than redisplay (in lisp, or at
> least not in redisplay) but still more or less transparent.

I have come to the conclusion that extending the underlying data
structure for the storage is unavoidable and, really, the simplest +
most efficient method to achieve transclusion / islands / seamless
mode regions / buffer lenses / whatever, and to avoid having to deal
with a whole lotta bugs, corner cases and synchronization trickery.

Emacs uses gap buffer for storing text (a c array with some semantics,
basically). What I have in mind is to keep it that way, but to have a
list of references to other buffers along with their intended
positions.

From what I saw, there's not a unified core interface on top of that c
array. So, would need to modify each place that makes a use of it.
(for a couple of examples, see text_outside_line_unchanged_p in
xdisp.c, looking_at_1 in search.c)

I tried to eyeball the effected landscape by looking at where GPT
(position of the gap) is referenced. There are 287 occurences of GPT
in the src directory, most of them are in buffer.c, buffer.h,
insdel.c. The distribution is: coding.c(24), syntax.c(10),
changelog(17), xdisp(8), insdel(131), buffer(35),
editfns/fns/w32xfns(11), search(7), marker(6), xml(1), charset(3),
composite(3), decompress(1), pdumper(1), process(1), character(3),
json(1), fileio(11), indent(3), sysdep(2).

The good news is: I believe xdisp.c wouldn't have to be modified all
that much if the regions are enforced to start on different lines.

This whole ordeal could easily take a few months. I, unfortunately, am
unlikely to find the time to do it this year.

PS A possibly useful outcome of this is the possibility to mitigate
extremely long gap motions by breaking up a big buffer into several
smaller ones.

пт, 10 апр. 2020 г. в 21:38, Ihor Radchenko <yantar92@gmail.com>:
>
> > Following mode plus atomic windows seems doable. But it wouldn’t be
> > transparent, AFAIK follow-mode needs to add special handlers for
> > things like isearch and query-replace. OTOH, it would be much more
> > difficult to implement in xdisp.c. I wonder if there is an
> > intermediate layer that’s higher level than redisplay (in lisp, or at
> > least not in redisplay) but still more or less transparent.
>
> What about extending the idea of indirect buffers? In indirect buffers,
> the buffer strings are associated with the same memory address storing
> the text (if I understand correctly). Similar thing (theoretically) can
> be done for individual segments of text. Indeed, there will still be a
> question on how the fontification is done, if the overlays should be
> shared, and how the key bindings should behave on such segments, but the
> basic functionality of automatically sharing text segments between
> buffers may be a good framework to start considering more complicated
> cases.
>
> Best,
> Ihor
>
> Yuan Fu <casouri@gmail.com> writes:
>
> > Following mode plus atomic windows seems doable. But it wouldn’t be transparent, AFAIK follow-mode needs to add special handlers for things like isearch and query-replace. OTOH, it would be much more difficult to implement in xdisp.c. I wonder if there is an intermediate layer that’s higher level than redisplay (in lisp, or at least not in redisplay) but still more or less transparent.
> >
> > Yuan
>
> --
> Ihor Radchenko,
> PhD,
> Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
> State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
> Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg

Re: Request for pointers and advice: displaying several buffers inside a single window

[chad]

[-- Attachment #1: Type: text/plain, Size: 4855 bytes --]

These are deep potential changes to emacs, but reading it today reminded me
of an article I read recently from the people who implement VS Code:


https://code.visualstudio.com/blogs/2018/03/23/text-buffer-reimplementation

This article describes a series of choices, revisions, and reasoning
leading up to VS Code's internal representation of the buffer text. In
particular, the way the found and added caching of line starts seemed like
something that could help emacs' problems with very long lines, and perhaps
also the concerns with parsing large files for highlighting and indenting.

~Chad

On Fri, Apr 10, 2020 at 12:10 PM Dmitrii Korobeinikov <dim1212k@gmail.com>
wrote:

> > Following mode plus atomic windows seems doable. But it wouldn’t be
> > transparent, AFAIK follow-mode needs to add special handlers for
> > things like isearch and query-replace. OTOH, it would be much more
> > difficult to implement in xdisp.c. I wonder if there is an
> > intermediate layer that’s higher level than redisplay (in lisp, or at
> > least not in redisplay) but still more or less transparent.
>
> I have come to the conclusion that extending the underlying data
> structure for the storage is unavoidable and, really, the simplest +
> most efficient method to achieve transclusion / islands / seamless
> mode regions / buffer lenses / whatever, and to avoid having to deal
> with a whole lotta bugs, corner cases and synchronization trickery.
>
> Emacs uses gap buffer for storing text (a c array with some semantics,
> basically). What I have in mind is to keep it that way, but to have a
> list of references to other buffers along with their intended
> positions.
>
> From what I saw, there's not a unified core interface on top of that c
> array. So, would need to modify each place that makes a use of it.
> (for a couple of examples, see text_outside_line_unchanged_p in
> xdisp.c, looking_at_1 in search.c)
>
> I tried to eyeball the effected landscape by looking at where GPT
> (position of the gap) is referenced. There are 287 occurences of GPT
> in the src directory, most of them are in buffer.c, buffer.h,
> insdel.c. The distribution is: coding.c(24), syntax.c(10),
> changelog(17), xdisp(8), insdel(131), buffer(35),
> editfns/fns/w32xfns(11), search(7), marker(6), xml(1), charset(3),
> composite(3), decompress(1), pdumper(1), process(1), character(3),
> json(1), fileio(11), indent(3), sysdep(2).
>
> The good news is: I believe xdisp.c wouldn't have to be modified all
> that much if the regions are enforced to start on different lines.
>
> This whole ordeal could easily take a few months. I, unfortunately, am
> unlikely to find the time to do it this year.
>
> PS A possibly useful outcome of this is the possibility to mitigate
> extremely long gap motions by breaking up a big buffer into several
> smaller ones.
>
> пт, 10 апр. 2020 г. в 21:38, Ihor Radchenko <yantar92@gmail.com>:
> >
> > > Following mode plus atomic windows seems doable. But it wouldn’t be
> > > transparent, AFAIK follow-mode needs to add special handlers for
> > > things like isearch and query-replace. OTOH, it would be much more
> > > difficult to implement in xdisp.c. I wonder if there is an
> > > intermediate layer that’s higher level than redisplay (in lisp, or at
> > > least not in redisplay) but still more or less transparent.
> >
> > What about extending the idea of indirect buffers? In indirect buffers,
> > the buffer strings are associated with the same memory address storing
> > the text (if I understand correctly). Similar thing (theoretically) can
> > be done for individual segments of text. Indeed, there will still be a
> > question on how the fontification is done, if the overlays should be
> > shared, and how the key bindings should behave on such segments, but the
> > basic functionality of automatically sharing text segments between
> > buffers may be a good framework to start considering more complicated
> > cases.
> >
> > Best,
> > Ihor
> >
> > Yuan Fu <casouri@gmail.com> writes:
> >
> > > Following mode plus atomic windows seems doable. But it wouldn’t be
> transparent, AFAIK follow-mode needs to add special handlers for things
> like isearch and query-replace. OTOH, it would be much more difficult to
> implement in xdisp.c. I wonder if there is an intermediate layer that’s
> higher level than redisplay (in lisp, or at least not in redisplay) but
> still more or less transparent.
> > >
> > > Yuan
> >
> > --
> > Ihor Radchenko,
> > PhD,
> > Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
> > State Key Laboratory for Mechanical Behavior of Materials, Xi'an
> Jiaotong University, Xi'an, China
> > Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg
>
>

[-- Attachment #2: Type: text/html, Size: 5845 bytes --]

Re: Request for pointers and advice: displaying several buffers inside a single window

[Eli Zaretskii]

> From: chad <yandros@gmail.com>
> Date: Fri, 10 Apr 2020 17:05:00 -0700
> Cc: Ihor Radchenko <yantar92@gmail.com>, Adam Porter <adam@alphapapa.net>, Yuan Fu <casouri@gmail.com>, 
> 	Eli Zaretskii <eliz@gnu.org>, emacs-devel <emacs-devel@gnu.org>
> 
> In particular, the way the found and added caching of line starts seemed like
> something that could help emacs' problems with very long lines, and perhaps also the concerns with parsing
> large files for highlighting and indenting.

I don't think so, because finding the next newline is not the
expensive part of the Emacs redisplay when lines are very long.  In
fact, we look for newlines by calling the memchr library function, and
the result is lightning-fast, because Emacs represents buffer text as
linear array of bytes, and memchr expands to a small number of machine
instructions in an optimized build.  With today's fast CPU clocks,
searching for the next newline in even a very large buffer is a
non-issue.

The expensive part of redisplay is finding the correspondence between
buffer positions and screen coordinates.  Every display update needs
some operation which has to find such correspondence.  For example,
take the simple C-n key, which by default moves to the next visual
line.  Long lines are normally wrapped on display, so the character
below the current one on the next visual line is likely to be some
place in the same physical line, but where exactly is that?  What
Emacs does internally is traverse all the characters after the current
one, calculating their metrics and making pixel-level layout
decisions, including the line wrapping, until it gets to the character
whose screen coordinates indicate it is directly below the current
cursor position.  Then Emacs puts point at that place.  C-p is
similar, but more expensive, because moving back requires to start
from the beginning of a physical line (which might be far away).  C-v
and M-v are also similar, because they need to calculate where to put
the window-start to display the next screen-full of text with some
overlap with what is currently shown.  Etc. etc. -- it is this task of
finding what buffer position corresponds to certain screen coordinates
is what makes redisplay painfully slow when there are very long lines.
Emacs before v21 supported only fixed-pitch fonts, where such
calculations were much easier, but we nowadays support variables fonts
in the same buffer.

Re: Request for pointers and advice: displaying several buffers inside a single window

[Eli Zaretskii]

> From: Dmitrii Korobeinikov <dim1212k@gmail.com>
> Date: Sat, 11 Apr 2020 01:09:08 +0600
> Cc: Yuan Fu <casouri@gmail.com>, adam@alphapapa.net, Eli Zaretskii <eliz@gnu.org>, 
> 	emacs-devel <emacs-devel@gnu.org>
> 
> >From what I saw, there's not a unified core interface on top of that c
> array. So, would need to modify each place that makes a use of it.
> (for a couple of examples, see text_outside_line_unchanged_p in
> xdisp.c, looking_at_1 in search.c)

I'm not sure why you came to that conclusion.  Emacs's internals
access buffer text directly when needed, for speed.  Interfaces like
BEG_UNCHANGED/END_UNCHANGED, BYTE_POS_ADDR, GPT, and others are no
less "core interfaces" for accessing buffer text than functions in
insdel.c.  I would even consider insdel.c as a layer _above_ the
low-level interfaces such as BYTE_POS_ADDR.

And looking_at_1 does what it does because it needs to pass buffer
text as 2 C 'char' arrays to regex routines, so I don't think it's any
evidence to anything.  It's just a trick to pass non-contiguous text
to a function that needs contiguous text.

> The good news is: I believe xdisp.c wouldn't have to be modified all
> that much if the regions are enforced to start on different lines.

How so?  The display code accesses the buffer text directly, using
BYTE_POS_ADDR.

Re: Request for pointers and advice: displaying several buffers inside a single window

[Dmitrii Korobeinikov]

> > From what I saw, there's not a unified core interface on top of that c
> > array. So, would need to modify each place that makes a use of it.
> > (for a couple of examples, see text_outside_line_unchanged_p in
> > xdisp.c, looking_at_1 in search.c)
>
> I'm not sure why you came to that conclusion.  Emacs's internals
> access buffer text directly when needed, for speed.  Interfaces like
> BEG_UNCHANGED/END_UNCHANGED, BYTE_POS_ADDR, GPT, and others are no
> less "core interfaces" for accessing buffer text than functions in
> insdel.c.  I would even consider insdel.c as a layer _above_ the
> low-level interfaces such as BYTE_POS_ADDR.

By "unified core interface", I meant something like an OOP-style
public interface done in a way so the users don't have to know about
the implementation and underlying data structures, in which case
modifications could've been simpler. I am not saying that's what it
should have been or even could have been, just stating the fact that a
different approach is used.

> Emacs's internals
> access buffer text directly when needed, for speed.

That's exactly the point I was making.

> > The good news is: I believe xdisp.c wouldn't have to be modified all
> > that much if the regions are enforced to start on different lines.
>
> How so?  The display code accesses the buffer text directly, using
> BYTE_POS_ADDR.

I didn't say it wouldn't have to be touched at all : ) xdisp.c has
only 8 occurences of BYTE_POS_ADDR, that doesn't appear too bad.

BTW What I am not sure about is if "different lines" enforcement would
make for a noticeably easier implementation, now I am thinking, it
probably wouldn't matter that much.

сб, 11 апр. 2020 г. в 13:30, Eli Zaretskii <eliz@gnu.org>:
>
> > From: Dmitrii Korobeinikov <dim1212k@gmail.com>
> > Date: Sat, 11 Apr 2020 01:09:08 +0600
> > Cc: Yuan Fu <casouri@gmail.com>, adam@alphapapa.net, Eli Zaretskii <eliz@gnu.org>,
> >       emacs-devel <emacs-devel@gnu.org>
> >
> > >From what I saw, there's not a unified core interface on top of that c
> > array. So, would need to modify each place that makes a use of it.
> > (for a couple of examples, see text_outside_line_unchanged_p in
> > xdisp.c, looking_at_1 in search.c)
>
> I'm not sure why you came to that conclusion.  Emacs's internals
> access buffer text directly when needed, for speed.  Interfaces like
> BEG_UNCHANGED/END_UNCHANGED, BYTE_POS_ADDR, GPT, and others are no
> less "core interfaces" for accessing buffer text than functions in
> insdel.c.  I would even consider insdel.c as a layer _above_ the
> low-level interfaces such as BYTE_POS_ADDR.
>
> And looking_at_1 does what it does because it needs to pass buffer
> text as 2 C 'char' arrays to regex routines, so I don't think it's any
> evidence to anything.  It's just a trick to pass non-contiguous text
> to a function that needs contiguous text.
>
> > The good news is: I believe xdisp.c wouldn't have to be modified all
> > that much if the regions are enforced to start on different lines.
>
> How so?  The display code accesses the buffer text directly, using
> BYTE_POS_ADDR.

Re: Request for pointers and advice: displaying several buffers inside a single window

[Eli Zaretskii]

> From: Dmitrii Korobeinikov <dim1212k@gmail.com>
> Date: Sat, 11 Apr 2020 13:56:07 +0600
> Cc: Ihor Radchenko <yantar92@gmail.com>, Yuan Fu <casouri@gmail.com>, adam@alphapapa.net, 
> 	emacs-devel <emacs-devel@gnu.org>
> 
> By "unified core interface", I meant something like an OOP-style
> public interface done in a way so the users don't have to know about
> the implementation and underlying data structures

You will find that in buffer.h.  Modify those interfaces, and you are
done.

> > > The good news is: I believe xdisp.c wouldn't have to be modified all
> > > that much if the regions are enforced to start on different lines.
> >
> > How so?  The display code accesses the buffer text directly, using
> > BYTE_POS_ADDR.
> 
> I didn't say it wouldn't have to be touched at all : ) xdisp.c has
> only 8 occurences of BYTE_POS_ADDR, that doesn't appear too bad.

Why does it matter how many times BYTE_POS_ADDR is used?  If you
change the implementation of BYTE_POS_ADDR, the replacement is a
mechanical job, no matter if there are 8 places or 800 to make the
change.

Re: Request for pointers and advice: displaying several buffers inside a single window

[Dmitrii Korobeinikov]

> Why does it matter how many times BYTE_POS_ADDR is used?  If you
> change the implementation of BYTE_POS_ADDR, the replacement is a
> mechanical job, no matter if there are 8 places or 800 to make the
> change.

True, but BYTE_POS_ADDR turns out to be a bad example and not a good
estimator of the needed changes at all, since it's a part of the
interface. I only now looked and saw it was a function and not a
field.

But, for instance, GPT is, because extra logic needs to be written to
accomodate multiple GPTs (one for the buffer, and one for each of the
referenced buffers recursively).

To give an example, search.c has this lines:

> p1 = BEGV_ADDR;
> s1 = GPT_BYTE - BEGV_BYTE;
> p2 = GAP_END_ADDR;
> s2 = ZV_BYTE - GPT_BYTE;

This would need to turn into an array of n boundaries, instead of 2.
And re_match_2 would need to become a re_match_n.

Next, indent.c L362:

>  else if (PT <= GPT || BEGV > GPT)

But there isn't just one GPT now, so, expect changes around this place as well.

I didn't look at the code very thoroughly, but I am pretty sure other
places aren't too different. Besides GPT, obviously, boundaries of the
referenced buffers now have to be accounted for, with their contents.
A supposedly ordinary loop job, but still. All of these changes are
conceptually simple, but do involve a bunch of legwork.

There are some exceptions of course. Like "where to insert a character
at the boundary", mentioned above. One natural solution for this, by
the way, could be to simply look at the previous position of the
point, i.e. if the boundary was approached from the right, insert to
the rightmost buffer.

сб, 11 апр. 2020 г. в 14:26, Eli Zaretskii <eliz@gnu.org>:
>
> > From: Dmitrii Korobeinikov <dim1212k@gmail.com>
> > Date: Sat, 11 Apr 2020 13:56:07 +0600
> > Cc: Ihor Radchenko <yantar92@gmail.com>, Yuan Fu <casouri@gmail.com>, adam@alphapapa.net,
> >       emacs-devel <emacs-devel@gnu.org>
> >
> > By "unified core interface", I meant something like an OOP-style
> > public interface done in a way so the users don't have to know about
> > the implementation and underlying data structures
>
> You will find that in buffer.h.  Modify those interfaces, and you are
> done.
>
> > > > The good news is: I believe xdisp.c wouldn't have to be modified all
> > > > that much if the regions are enforced to start on different lines.
> > >
> > > How so?  The display code accesses the buffer text directly, using
> > > BYTE_POS_ADDR.
> >
> > I didn't say it wouldn't have to be touched at all : ) xdisp.c has
> > only 8 occurences of BYTE_POS_ADDR, that doesn't appear too bad.
>
> Why does it matter how many times BYTE_POS_ADDR is used?  If you
> change the implementation of BYTE_POS_ADDR, the replacement is a
> mechanical job, no matter if there are 8 places or 800 to make the
> change.

Re: [SPAM UNSURE] Request for pointers and advice: displaying several buffers inside a single window

[Stephen Leake]

Dmitrii Korobeinikov <dim1212k@gmail.com> writes:

> Hi all,
>
> I want to explore the feasibility of displaying two (indirect) buffers
> inside a single window, one after another. Each buffer has to look
> like it's rendered in its own mode. For the starters, each buffer can
> start on its own new line in the window. No interaction w/ text is
> required for now.

This would look the same as two windows each displaying one buffer, if
you could eliminate the mode line in betweeen them. That should be much
easier.

Hmm. Currently, the bottom line in a window is only partially displayed.
So you'd also have to adjust the top window height to allow displaying
the full last line. Or not; depends on what you want.

-- 
-- Stephe

Re: [SPAM UNSURE] Request for pointers and advice: displaying several buffers inside a single window

[Robert Pluim]

>>>>> On Fri, 03 Apr 2020 10:30:38 -0800, Stephen Leake <stephen_leake@stephe-leake.org> said:

    Stephen> Dmitrii Korobeinikov <dim1212k@gmail.com> writes:
    >> Hi all,
    >> 
    >> I want to explore the feasibility of displaying two (indirect) buffers
    >> inside a single window, one after another. Each buffer has to look
    >> like it's rendered in its own mode. For the starters, each buffer can
    >> start on its own new line in the window. No interaction w/ text is
    >> required for now.

    Stephen> This would look the same as two windows each displaying one buffer, if
    Stephen> you could eliminate the mode line in betweeen them. That should be much
    Stephen> easier.

(setq mode-line-format nil)

and then make the second window display an indirect buffer to the buffer
displayed in the first window, so you can change the major mode, then
enable follow-mode so that scrolling one scrolls the other. Untested :-)

    Stephen> Hmm. Currently, the bottom line in a window is only partially displayed.
    Stephen> So you'd also have to adjust the top window height to allow displaying
    Stephen> the full last line. Or not; depends on what you want.

I think that only happens if you have one or more of the various
*pixelwise* variables non-nil; the default is to do everything in
multiples of line-height.

Robert

Re: [SPAM UNSURE] Request for pointers and advice: displaying several buffers inside a single window

[Dmitrii Korobeinikov]

I don't know why, but I never received the email from Stephen in my
inbox (Robert's I have). I replied via "reply via e-mail to" button on
the tracker, but my reply was filed into a seperate thread [1]. I
wonder how not to make that happen again...

But, first of all, sorry, I think I have miscommunicated my general
intent in the OP. What I meant to say is that in the end I will want
the two buffers behave as if they were two regions of the same master
buffer (implying, for instance, the possibility of point seamlessly
jumping across their borders), and I thought that ommiting the
interactivity part and hacking up the renderer would be a good start.
That, admittedly, wasn't very far-sighted of me. And now I am
beginning to embrace the inevitability of having to extend upon the
gap buffer.

[1] https://lists.gnu.org/archive/html/emacs-devel/2020-04/msg00260.html

вс, 5 апр. 2020 г. в 19:18, Robert Pluim <rpluim@gmail.com>:
>
> >>>>> On Fri, 03 Apr 2020 10:30:38 -0800, Stephen Leake <stephen_leake@stephe-leake.org> said:
>
>     Stephen> Dmitrii Korobeinikov <dim1212k@gmail.com> writes:
>     >> Hi all,
>     >>
>     >> I want to explore the feasibility of displaying two (indirect) buffers
>     >> inside a single window, one after another. Each buffer has to look
>     >> like it's rendered in its own mode. For the starters, each buffer can
>     >> start on its own new line in the window. No interaction w/ text is
>     >> required for now.
>
>     Stephen> This would look the same as two windows each displaying one buffer, if
>     Stephen> you could eliminate the mode line in betweeen them. That should be much
>     Stephen> easier.
>
> (setq mode-line-format nil)
>
> and then make the second window display an indirect buffer to the buffer
> displayed in the first window, so you can change the major mode, then
> enable follow-mode so that scrolling one scrolls the other. Untested :-)
>
>     Stephen> Hmm. Currently, the bottom line in a window is only partially displayed.
>     Stephen> So you'd also have to adjust the top window height to allow displaying
>     Stephen> the full last line. Or not; depends on what you want.
>
> I think that only happens if you have one or more of the various
> *pixelwise* variables non-nil; the default is to do everything in
> multiples of line-height.
>
> Robert