[Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)

[Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)

[Dmitrii Korobeinikov]

[-- Attachment #1: Type: text/plain, Size: 218 bytes --]

I have written a proposal for buffer lenses which could prove useful in
Org-mode, especially for interacting with code.
If you are interested, please, see this link:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=35419

[-- Attachment #2: Type: text/html, Size: 374 bytes --]

Re: [Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 1736 bytes --]

Dear Dmitrii,

I strongly support the proposal.

Another use case for me is to speed up agenda creation.
I usually do not like to split my org files into too many. However, it
results in very large and slow org buffers later. If I can store some
parts of the org files externally and only show them if some condition
is met (say, for certain todo state of the parent entry), it would speed
up my agenda and the buffer navigation quite significantly.

Example:
#+begin_src org
* Projects
** 2019
*** TODO Project 1     :ORG:
# the project contents is stored in an external file
:PROPERTIES:
:ORG-FILE: project1.org
:END:
# beginning of a lense, which is linked to project1.org
**** Heading 1
**** Heading 2
And many headings below
# ...
# end of the lense
*** HOLD Project 2     :ORG:
:PROPERTIES:
:ORG-FILE: project2.org
:END:
# beginning of another lense
# nothing is included here because the project state is =HOLD=
# end of the lense
#+end_src

Let me put some historical context to this proposal.
There was a discussion of similar feature in emacs-dev last year.
The idea was to implement nested buffers:
https://lists.gnu.org/archive/html/emacs-devel/2018-07/msg00863.html 

There are also several projects, which implement part of the
functionality you described:
- mmm-mode: https://github.com/purcell/mmm-mode
- polymode: https://github.com/polymode/polymode

Best,
Ihor

Dmitrii Korobeinikov <dim1212k@gmail.com> writes:

> I have written a proposal for buffer lenses which could prove useful in
> Org-mode, especially for interacting with code.
> If you are interested, please, see this link:
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=35419

-- 
Ihor Radchenko,

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#35419: [Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)

[Noam Postavsky]

Dmitrii Korobeinikov <dim1212k@gmail.com> writes:

> * Implementation
>
>   I am not familiar with Emacs internals to say what's feasible of the
> proposed structure.

Have you looked at Phil Lord's lentic package?  I think it implements a
lot of what you're talking about.

https://github.com/phillord/lentic

bug#35419: Fwd: Re: [Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)

['Ihor Radchenko']

[-- Attachment #1.1: Type: message/rfc822, Size: 2972 bytes --]

[-- Attachment #1.1.1.1: Type: text/plain, Size: 1736 bytes --]

Dear Dmitrii,

I strongly support the proposal.

Another use case for me is to speed up agenda creation.
I usually do not like to split my org files into too many. However, it
results in very large and slow org buffers later. If I can store some
parts of the org files externally and only show them if some condition
is met (say, for certain todo state of the parent entry), it would speed
up my agenda and the buffer navigation quite significantly.

Example:
#+begin_src org
* Projects
** 2019
*** TODO Project 1     :ORG:
# the project contents is stored in an external file
:PROPERTIES:
:ORG-FILE: project1.org
:END:
# beginning of a lense, which is linked to project1.org
**** Heading 1
**** Heading 2
And many headings below
# ...
# end of the lense
*** HOLD Project 2     :ORG:
:PROPERTIES:
:ORG-FILE: project2.org
:END:
# beginning of another lense
# nothing is included here because the project state is =HOLD=
# end of the lense
#+end_src

Let me put some historical context to this proposal.
There was a discussion of similar feature in emacs-dev last year.
The idea was to implement nested buffers:
https://lists.gnu.org/archive/html/emacs-devel/2018-07/msg00863.html 

There are also several projects, which implement part of the
functionality you described:
- mmm-mode: https://github.com/purcell/mmm-mode
- polymode: https://github.com/polymode/polymode

Best,
Ihor

Dmitrii Korobeinikov <dim1212k@gmail.com> writes:

> I have written a proposal for buffer lenses which could prove useful in
> Org-mode, especially for interacting with code.
> If you are interested, please, see this link:
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=35419

-- 
Ihor Radchenko,

[-- Attachment #1.1.1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

[-- Attachment #1.2: Type: text/plain, Size: 24 bytes --]


-- 
Ihor Radchenko,

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#35419: [Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)

[Dmitrii Korobeinikov]

[-- Attachment #1: Type: text/plain, Size: 1287 bytes --]

> Have you looked at Phil Lord's lentic package?  I think it implements a
> lot of what you're talking about.

> https://github.com/phillord/lentic

This is nice to see!
Indeed, except for embedding, there is a large overlap with what I
described as buffer lenses.

BTW, judging by this description: "changes percolation now happens
incrementally, so only those parts of the buffer are updated. As a result,
lentic now cope with long files with little noticable delay", the buffers
don't share any data and need to sync with the master [linked] buffer.
Is this the best solution? I have imagined that at the low level there is
an actual data structure that keeps the raw textual data and it could be
directly shared by multiple buffers. I mean, when a buffer is saved to a
file, the text doesn't need to be stripped of properties beforehand, right?

чт, 25 апр. 2019 г. в 07:37, Noam Postavsky <npostavs@gmail.com>:

> Dmitrii Korobeinikov <dim1212k@gmail.com> writes:
>
> > * Implementation
> >
> >   I am not familiar with Emacs internals to say what's feasible of the
> > proposed structure.
>
> Have you looked at Phil Lord's lentic package?  I think it implements a
> lot of what you're talking about.
>
> https://github.com/phillord/lentic
>

[-- Attachment #2: Type: text/html, Size: 1957 bytes --]

bug#35419: [Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)

[Philipp Stephani]

Am Do., 25. Apr. 2019 um 10:41 Uhr schrieb Dmitrii Korobeinikov
<dim1212k@gmail.com>:
> I have imagined that at the low level there is an actual data structure that keeps the raw textual data and it could be directly shared by multiple buffers.

That's what indirect buffers do. Maybe the indirect buffer
functionality could be beefed up to support what you want?

bug#35419: [Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)

[Dmitrii Korobeinikov]

[-- Attachment #1: Type: text/plain, Size: 1153 bytes --]

чт, 25 апр. 2019 г. в 23:52, Philipp Stephani <p.stephani2@gmail.com>:

> Am Do., 25. Apr. 2019 um 10:41 Uhr schrieb Dmitrii Korobeinikov
> <dim1212k@gmail.com>:
> > I have imagined that at the low level there is an actual data structure
> that keeps the raw textual data and it could be directly shared by multiple
> buffers.
>
> That's what indirect buffers do. Maybe the indirect buffer
> functionality could be beefed up to support what you want?
>

https://www.gnu.org/software/emacs/manual/html_node/emacs/Indirect-Buffers.html
> The text of the indirect buffer is always identical to the text of its
base buffer; changes made by editing either one are visible immediately in
the other. But in all other respects, the indirect buffer and its base
buffer are completely separate. They can have different names, different
values of point, different narrowing, different markers, different major
modes, and different local variables.

Awesome! Looks like we have some solid rails to drive on.

BTW what's the purpose of lentic-mode then? To be "providing multiple
persistent views"?
https://github.com/phillord/lentic

[-- Attachment #2: Type: text/html, Size: 1758 bytes --]

Re: bug#35419: [Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)

[Roland Everaert]

I see lens to be useful for the eev mode, too.

Roland.

Dmitrii Korobeinikov writes:

>> Have you looked at Phil Lord's lentic package?  I think it implements a
>> lot of what you're talking about.
>
>> https://github.com/phillord/lentic
>
> This is nice to see!
> Indeed, except for embedding, there is a large overlap with what I
> described as buffer lenses.
>
> BTW, judging by this description: "changes percolation now happens
> incrementally, so only those parts of the buffer are updated. As a result,
> lentic now cope with long files with little noticable delay", the buffers
> don't share any data and need to sync with the master [linked] buffer.
> Is this the best solution? I have imagined that at the low level there is
> an actual data structure that keeps the raw textual data and it could be
> directly shared by multiple buffers. I mean, when a buffer is saved to a
> file, the text doesn't need to be stripped of properties beforehand, right?
>
> чт, 25 апр. 2019 г. в 07:37, Noam Postavsky <npostavs@gmail.com>:
>
>> Dmitrii Korobeinikov <dim1212k@gmail.com> writes:
>>
>> > * Implementation
>> >
>> >   I am not familiar with Emacs internals to say what's feasible of the
>> > proposed structure.
>>
>> Have you looked at Phil Lord's lentic package?  I think it implements a
>> lot of what you're talking about.
>>
>> https://github.com/phillord/lentic
>>


-- 
Luke, use the FOSS

Sent from Emacs

bug#35419: bug#35419: [Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)

[Dmitrii Korobeinikov]

[-- Attachment #1: Type: text/plain, Size: 314 bytes --]

> I see lens to be useful for the eev mode, too.

Never heard of eev, but judging by some demos, it's a way to execute elisp
commands interactively.
Something like stitching blocks of commands together, or the data to
operate on, or embedding a target such as a shell in the same buffer is the
use-case idea then?

[-- Attachment #2: Type: text/html, Size: 503 bytes --]

bug#35419: [Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)

[Dmitrii Korobeinikov]

[-- Attachment #1: Type: text/plain, Size: 1702 bytes --]

I found a clarification on how mmm-mode works.

https://github.com/polymode/polymode/issues/187
> mmm-mode also allows having multiple major modes depending on cursor
position in the buffer. However, it does not fully replace major mode
locally. This mode is only taking care about keymap, menu, local variables,
font-lock, and indentation. It does not really take care about the minor
modes and does not run the submode hooks either.

Just to reiterate, polymode's idea is to switch between indirect buffers,
one for each major mode.

OK, detail largely disregarded, I now can draw a bird-eye view comparison
between lenses and multi-mode modes.

- Neither polymode nor mmm-mode treat a region as if it were truly on its
own in a seperate buffer.

Effects: no stuff like seperate truncation options, implied syntax checking
and so on.

- Moreover, the region must be a part of the buffer.

Effects: no data sharing between buffers, no possibility of stitching
different buffers together, etc.

Now, with these out of the way.

Indirect buffers give the answer to the issue of sharing some textual data
between several buffer.
(1) A question: when an indirect buffer is created and some region is
narrowed to, is the rest of the buffer duplicated in memory somewhere? If
this is so, there could be a useful efficiency-related modification to
indirect buffers, which would allow "hard-narrowing": not duplicating the
rest of the base buffer.

The next immediately outstanding question is:
(2) how can "embedding" (of a buffer as a part of another buffer as an
area) be done efficiently? This could possibly be approached as two
problems: (i) displaying the area and (ii) interacting with it.
Any ideas?

[-- Attachment #2: Type: text/html, Size: 2127 bytes --]

bug#35419: [Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)

[Dmitrii Korobeinikov]

[-- Attachment #1: Type: text/plain, Size: 3017 bytes --]

Dear Ihor,

> Another use case for me is to speed up agenda creation.
> I usually do not like to split my org files into too many. However, it
> results in very large and slow org buffers later. If I can store some
> parts of the org files externally and only show them if some condition
> is met (say, for certain todo state of the parent entry), it would speed
> up my agenda and the buffer navigation quite significantly.

That's a good one!

> Let me put some historical context to this proposal.
> There was a discussion of similar feature in emacs-dev last year.
> The idea was to implement nested buffers:
> https://lists.gnu.org/archive/html/emacs-devel/2018-07/msg00863.html

An interesting read, provides another use-case (collect external data in
one place to easily view/edit):
https://lists.gnu.org/archive/html/emacs-devel/2018-07/msg00890.html

> There are also several projects, which implement part of the
> functionality you described:
> - mmm-mode: https://github.com/purcell/mmm-mode
> - polymode: https://github.com/polymode/polymode

Pretty cool stuff. For thoroughness, let's discuss how these work.

I found a comment which mentions polymode's working principle.
https://www.reddit.com/r/emacs/comments/50p34n/polymode_is_awesome/?depth=1
>> Polymode doesn't keep its modes in a single emacs buffer but in several
indirect buffers, as many as different modes are there in a file.
Consequently, polymode is as fast as switching emacs buffers because it
never re-installs major modes like other multi-modes do. Dave Love's
multi-mode.el gets full credit for this idea.
> It looks like it slows emacs to a crawl in my main org config file. It
seems to work fairly well in some of my notes files (though with some weird
indenting behavior).

Basically, simplicity is in place but at the cost of duplication.
Lenses could avoid duplication, while yielding increased functionality and
speed.
(e.g. in polymode, a syntax checker couldn't yield correct results unless
narrowing was constantly used, which is inefficient)

Now, to MMM-mode. According to the info file:

> Within the file, MMM-mode creates /submode regions/ within which other
major modes are in effect.

> While the point is in a submode region, the following changes occur:
> <...> keymap <...> local variables <...> syntax table and indentation
<...> font-lock

> The submode regions are represented internally by Emacs Lisp objects
known as /overlays/.

> A lot of the functionality of MMM Mode---that which makes the major mode
> appear to change---is implemented by saving and restoring the values of
> local variables, or pseudo-variables.

What I don't understand is where the modes of the submode region run and
when they are turned on.
Are necessary modes just allowed to run at the right time for the whole
buffer? But then, how are they limited in their effect to just the
necessary region? Narrowing?
Could, for example, syntax checking be done efficiently that way?
Could someone, please, explain?

Best regards,
Dmitrii.

[-- Attachment #2: Type: text/html, Size: 4175 bytes --]

bug#35419: [O] [Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)

[Dmitry Gutov]

Hi!

Some late clarifications about mmm-mode.

On 26.04.2019 00:00, Dmitrii Korobeinikov wrote:
>  > A lot of the functionality of MMM Mode---that which makes the major mode
>  > appear to change---is implemented by saving and restoring the values of
>  > local variables, or pseudo-variables.
> 
> What I don't understand is where the modes of the submode region run and 
> when they are turned on.

They are run in an empty temporary buffer, see mmm-update-mode-info. 
That is true for all the "submodes" in a buffer. The primary major mode 
is run in the context of that buffer (IIRC). After any of them runs, the 
code responsible for it collects the values of a certain number of known 
variables and associates that map with the major mode (this is a bit of 
a simplification).

> Are necessary modes just allowed to run at the right time for the whole 
> buffer?

When you move between the "chunks", no major mode functions are called. 
Instead, the values of variables are swapped in. Including the value of 
the 'major-mode' variable.

> But then, how are they limited in their effect to just the 
> necessary region? Narrowing?

Usually, yes. Especially when we're talking about font-lock and 
syntax-propertize-function. See mmm-fontify-region-list for an example.

> Could, for example, syntax checking be done efficiently that way?

That depends on the combination of modes and how they are used (either 
they can be nested, like in web templates, or it's a flat list where 
chunks are largely independent like in Jupyter). But in most cases, I 
think, you could pick a good strategy. There are no universal ones, though.

bug#35419: [O] [Proposal] Buffer Lenses and the Case of Org-Mode (also, Jupyter)

[Dmitrii Korobeinikov]

Thank you for the insight and the references! Quite useful to learn
about this stuff.

вс, 5 апр. 2020 г. в 07:46, Dmitry Gutov <dgutov@yandex.ru>:
>
> Hi!
>
> Some late clarifications about mmm-mode.
>
> On 26.04.2019 00:00, Dmitrii Korobeinikov wrote:
> >  > A lot of the functionality of MMM Mode---that which makes the major mode
> >  > appear to change---is implemented by saving and restoring the values of
> >  > local variables, or pseudo-variables.
> >
> > What I don't understand is where the modes of the submode region run and
> > when they are turned on.
>
> They are run in an empty temporary buffer, see mmm-update-mode-info.
> That is true for all the "submodes" in a buffer. The primary major mode
> is run in the context of that buffer (IIRC). After any of them runs, the
> code responsible for it collects the values of a certain number of known
> variables and associates that map with the major mode (this is a bit of
> a simplification).
>
> > Are necessary modes just allowed to run at the right time for the whole
> > buffer?
>
> When you move between the "chunks", no major mode functions are called.
> Instead, the values of variables are swapped in. Including the value of
> the 'major-mode' variable.
>
> > But then, how are they limited in their effect to just the
> > necessary region? Narrowing?
>
> Usually, yes. Especially when we're talking about font-lock and
> syntax-propertize-function. See mmm-fontify-region-list for an example.
>
> > Could, for example, syntax checking be done efficiently that way?
>
> That depends on the combination of modes and how they are used (either
> they can be nested, like in web templates, or it's a flat list where
> chunks are largely independent like in Jupyter). But in most cases, I
> think, you could pick a good strategy. There are no universal ones, though.