trunk r116426: * lisp/jit-lock.el (jit-lock-mode): Keep it disabled in indirect buffers.

trunk r116426: * lisp/jit-lock.el (jit-lock-mode): Keep it disabled in indirect buffers.

[João Távora]

Hi,

I just read

   http://demonastery.org/2013/04/emacs-narrow-to-region-indirect/

which is a great demonstration of the utility of combining two existing
emacs core features: region-narrowing and indirect buffers.

In the comments, it goes on to suggest (as does the screen-shot) that
this could be a good replacement for the multiple-major-mode holy
grail. Someone here also asked about some kind of mini-editing-windows
after Adobe Brackets and I also think this could be a good alternative.

Problem is, that when I to switch modes in an indirect buffer, I get
this:

    "Not enabling jit-lock: it does not work in indirect buffer"

Then, no fontification and subsequent disappointment.

I don't know anything about fontification techniques, so I don't even
know if the question makes sense, but isn't there a way to get
fontification (even if slower than jit-lock-mode) in the indirect
buffer, and keep the base buffer jit-locked?

João

Re: trunk r116426: * lisp/jit-lock.el (jit-lock-mode): Keep it disabled in indirect buffers.

[Glenn Morris]

(Using two mailing lists is almost never appropriate.)

Indirect buffers are not well regarded; see eg

http://lists.gnu.org/archive/html/help-gnu-emacs/2014-01/msg00571.html

Re: trunk r116426: * lisp/jit-lock.el (jit-lock-mode): Keep it disabled in indirect buffers.

[João Távora]

Glenn Morris <rgm@gnu.org> writes:

> (Using two mailing lists is almost never appropriate.)

I thought this could trigger different takes from both user-level (some
easy workaround) and development perspectives (the discussion you linked
to), so I cross posted, seemed appropriate, but sorry.

> Indirect buffers are not well regarded; see eg
>
> http://lists.gnu.org/archive/html/help-gnu-emacs/2014-01/msg00571.html

Thanks, that discussion seems exhaustive, will read up.

Re: trunk r116426: * lisp/jit-lock.el (jit-lock-mode): Keep it disabled in indirect buffers.

[Stefan Monnier]

> I don't know anything about fontification techniques, so I don't even
> know if the question makes sense, but isn't there a way to get
> fontification (even if slower than jit-lock-mode) in the indirect
> buffer, and keep the base buffer jit-locked?

Everything is possible, yes.  But I just recommend you don't go there,
it's a waste of time.


        Stefan

Re: trunk r116426: * lisp/jit-lock.el (jit-lock-mode): Keep it disabled in indirect buffers.

[Vitalie Spinu]

 >>> Stefan Monnier on Mon, 07 Apr 2014 16:40:19 -0400 wrote:

 >> I don't know anything about fontification techniques, so I don't even
 >> know if the question makes sense, but isn't there a way to get
 >> fontification (even if slower than jit-lock-mode) in the indirect
 >> buffer, and keep the base buffer jit-locked?

 SM> Everything is possible, yes.  But I just recommend you don't go there,
 SM> it's a waste of time.

Waste of time or not, it has been done: https://github.com/vitoshka/polymode

Indirect buffers constitute the core of mode switching but are not
crucial for the rest of the package. I intentionally abstracted it from
all other stuff. If emacs ever comes with something smarter than
indirect-buffers I would be happy to adapt the mode switching mechanism.

I have been using it for more than an year in an unfinished state for
markdown and noweb and the experience was pretty delightful. Indirect
buffers are very fast and one-mode-per-indirect-buffer strategy makes
the whole system pretty robust.

Only recently I found time to finish the design, but then discovered
that jit-lock was removed. I am piggybacking on jit-lock and its removal
is a real bad luck. The last thing I need right now is to re-implement
jit-lock in indirect buffers.

Would it be possible to add a binary switch so that people can activate
jit-lock if they really need it?

Thanks, 

   Vitalie

Re: trunk r116426: * lisp/jit-lock.el (jit-lock-mode): Keep it disabled in indirect buffers.

[Stefan Monnier]

> Waste of time or not, it has been done: https://github.com/vitoshka/polymode

How 'bout we add it to GNU ELPA?

> Only recently I found time to finish the design, but then discovered
> that jit-lock was removed.  I am piggybacking on jit-lock and its removal
> is a real bad luck.  The last thing I need right now is to re-implement
> jit-lock in indirect buffers.

jit-lock support was broken.  So if you want it back, we need to fix it.
The problem is simple: jit-lock relies on the `fontified' text-property
to decide what needs to be (re)fontified.  And like all text-properties,
this property is shared between the base buffer and the
indirect buffers.

So when jit-lock is triggered it has to refontify in all buffers that
share the same base buffer.  And if font-lock is activated in several
buffers which share the same base buffer, they'll fight over their
shared `face' text-property.

A related issue is that after/before-change-functions are currently only
run in the current-buffer, even though the changes affect all buffers
which share the same base-buffer.

I think the right fix is to change the C code such that
fontification-functions and after/before-change-functions are always
obeyed only in the base buffer.  I.e. when displaying an indirect
buffer, we'd check the fontification-functions of the base buffer and
run jit-lock in that base-buffer.  When making changes in an indirect
buffer, we'd check after/before-change-functions in the base buffer and
run them there.

Do you think you could try to write such a patch?


        Stefan

Re: trunk r116426: * lisp/jit-lock.el (jit-lock-mode): Keep it disabled in indirect buffers.

[Vitalie Spinu]

 >>> Stefan Monnier on Fri, 23 May 2014 09:27:35 -0400 wrote:

 >> Waste of time or not, it has been done: https://github.com/vitoshka/polymode
 > How 'bout we add it to GNU ELPA?

That would be great, but how often can I update it at ELPA? AFAIK elpa
is for "stable" packages and polymode is still pretty far from being
stable. There is still some work to be done - chunk cashing with text
properties, indentation tweaks, poly-web-mode, etc.  MELPA works just
great for quick syncs.

[...]

 > So when jit-lock is triggered it has to refontify in all buffers that
 > share the same base buffer.  And if font-lock is activated in several
 > buffers which share the same base buffer, they'll fight over their
 > shared `face' text-property.

I think I saw a couple of infloops because of that.

 > A related issue is that after/before-change-functions are currently only
 > run in the current-buffer, even though the changes affect all buffers
 > which share the same base-buffer.

 > I think the right fix is to change the C code such that
 > fontification-functions and after/before-change-functions are always
 > obeyed only in the base buffer.  I.e. when displaying an indirect
 > buffer, we'd check the fontification-functions of the base buffer and
 > run jit-lock in that base-buffer.  When making changes in an indirect
 > buffer, we'd check after/before-change-functions in the base buffer
 > and run them there.

Why exactly after/before-change-functions should work in the base buffer
only?

At least from polymode prospective it would be ideal if
font-lock/change-functions could act *only* on current buffer. In
polymode each mode has its own buffer and font-lock and change-functions
are very different in base and indirect buffers. So, if I am in C++
chunk, font-lock and change-functions better be from and act on current
C++ buffer, not on the base buffer which is typically in fundamental
mode.

I guess for after/before-change-functions this argument applies more
broadly. One might want to have a change-function that acts differently
depending on whether the current buffer is indirect or not. For example
auto complete popup implemented with overlays must act in current buffer
because overlays are not shared.

How about fixing the problem by enforcing the font-lock/change-functions
only in the current buffer? That is, when in indirect buffer, fontify
only indirect buffer, when in the base buffer, fontify only the base
buffer. I thought this was how it worked before.

 > Do you think you could try to write such a patch?

I am not familiar with emacs C internals but I can definitely look into
that; great opportunity to learn. It won't be very quick, though.

  Vitalie

Re: trunk r116426: * lisp/jit-lock.el (jit-lock-mode): Keep it disabled in indirect buffers.

[Stefan Monnier]

> That would be great, but how often can I update it at ELPA?

`elpa' is a Git repository, so you can hack there to your heart's content.

> AFAIK elpa is for "stable" packages

That's a common misconception.

> and polymode is still pretty far from being stable. There is still
> some work to be done - chunk cashing with text properties, indentation
> tweaks, poly-web-mode, etc.  MELPA works just great for quick syncs.

Currently, the scripts at elpa.gnu.org only build "stable" packages,
i.e. a new package is released on GNU ELPA only when you bump the
"Version:" header of your package in the `elpa' repository.

I'd like to extend this so people can also install "whatever is at the
head of the branch".  Currently, they can do it by checking out the
`elpa' branch and using "make" in it, which puts into into an
"installed" state (so just just need to add the directory to their
package-directory-list), but then they get not just that one package,
but *all* GNU ELPA packages.

We could change the elpa.gnu.org scripts so that they additionally build
a secondary archive that would be a sort of GNU MELPA.  Or we could
provide new commands (a la "quelpa") to install a package directly from
the `elpa' branch.

> Why exactly after/before-change-functions should work in the base buffer
> only?

It's the simplest option.  If you want them to apply to other buffers as
well, then you can still get that by adding
a after/before-change-function to the base buffer which runs the
after/before-change-functions in the indirect buffer.

> So, if I am in C++ chunk, font-lock and change-functions better be
> from and act on current C++ buffer, not on the base buffer which is
> typically in fundamental mode.

What happens if you're inside a HTML chunk which has an embedded PHP
chunk and you edit that PHP chunk from that HTML-mode buffer instead of
from the PHP-mode buffer?

> I guess for after/before-change-functions this argument applies more
> broadly. One might want to have a change-function that acts differently
> depending on whether the current buffer is indirect or not. For example
> auto complete popup implemented with overlays must act in current buffer
> because overlays are not shared.

Indeed, there are all kinds of possible scenarios, so the C code can't
know which set of options to choose.  Instead it has to choose one
option which is "safe" and from which your code can then do what needs
to be done.


        Stefan

Re: trunk r116426: * lisp/jit-lock.el (jit-lock-mode): Keep it disabled in indirect buffers.

[Vitalie Spinu]

 >>> Stefan Monnier on Fri, 23 May 2014 16:49:09 -0400 wrote:

[...]

 >> Why exactly after/before-change-functions should work in the base buffer
 >> only?

 > It's the simplest option.  If you want them to apply to other buffers
 > as well, then you can still get that by adding a
 > after/before-change-function to the base buffer which runs the
 > after/before-change-functions in the indirect buffer.

Note that if I am in the indirect buffer, I don't want to run
base-buffer after/before-change-functions at all! I only want the hooks
from the indirect buffer. So, what you propose is not quite that simple,
if possible at all. Each time the after/before-change-functions are
initialized in indirect-buffer I will have to somehow prohibit execution
of the base-buffer change-functions and execute those from
indirect-buffer. Kinda mind-stretch. That there is no way to figure out
indirect buffers from the base buffer complicates the stuff even
further.

More generally, the fact that hooks are not executed in current-buffer
will probably cause havoc in other usages of indirect buffers. I imagine
that all designers of after/before-change-functions assume that the
action happens in the current buffer. Your proposal will invalidate this
assumption.

In conclusion, unless I miss something essential, I don't see it as "the
simplest option". The simplest option is to execute the hooks only in
the buffer that initiated the hook (always current buffer?).

 >> So, if I am in C++ chunk, font-lock and change-functions better be
 >> from and act on current C++ buffer, not on the base buffer which is
 >> typically in fundamental mode.

 > What happens if you're inside a HTML chunk which has an embedded PHP
 > chunk and you edit that PHP chunk from that HTML-mode buffer instead of
 > from the PHP-mode buffer?

This is not possible by design. Whenever you move the point the buffer
is switched behind the scene. So if you are in PHP chunk you are in
php-mode buffer, if you are in HTML chunk you are in html-mode buffer.

[...]

 > Indeed, there are all kinds of possible scenarios, so the C code can't
 > know which set of options to choose.  

Same as above; I am confused. What scenarios do you mean concretely?

What can go wrong with "run hooks in the current buffer only"
implementation? C can simply check if the hook is executed in the
current buffer. If some after/before-change-function from an indirect
buffer wants to make a change to base-buffer, then it can explicitly do
so. This is way more straightforward than the other way around.

The idea of indirect buffers is great. They share only text and text
properties, otherwise they are completely independent entities. Running
indirect buffer's hooks in the base-buffer would create an unnecessary
functional linkage. Hence, defeating the original intent of the indirect
buffers and obfuscating the reasoning about the code.


  Vitalie

Re: trunk r116426: * lisp/jit-lock.el (jit-lock-mode): Keep it disabled in indirect buffers.

[Stefan Monnier]

> Note that if I am in the indirect buffer, I don't want to run
> base-buffer after/before-change-functions at all!

The problem is that you're not the only user of
after/before-change-functions.  In the general case, all the related
buffers might care to know that some part of their text has
been modified.

> Each time the after/before-change-functions are initialized in
> indirect-buffer I will have to somehow prohibit execution of the
> base-buffer change-functions and execute those from
> indirect-buffer.

Why prohibit the hooks from the base buffer?  If they're there it's
presumably because they're needed.

> Kinda mind-stretch. That there is no way to figure out indirect
> buffers from the base buffer complicates the stuff even further.

It's easy to find all the indirect buffers.  In your case you'll just
want to find the indirect buffer that corresponds to a particular buffer
position, and that's something I'd expect you need at other places anyway.

> More generally, the fact that hooks are not executed in current-buffer
> will probably cause havoc in other usages of indirect buffers. I imagine
> that all designers of after/before-change-functions assume that the
> action happens in the current buffer. Your proposal will invalidate this
> assumption.

Running the after/before-change-functions of the base buffer in the base
buffer means that "the action happens in the current buffer", and indeed
it does: a buffer-modification happens in all the related buffers, not
just in the one that happens to be current.

> In conclusion, unless I miss something essential, I don't see it as "the
> simplest option". The simplest option is to execute the hooks only in
> the buffer that initiated the hook (always current buffer?).

That means the other related buffers aren't told about the modification
in their text, which is also unexpected.

> What can go wrong with "run hooks in the current buffer only"
> implementation?

OK, let's say you chunks

    chunk 1 in mode A
    chunk 2 in mode B
    chunk 3 in mode A

And let's say that mode A does some clever caching, so it sets up an
after-change-functions hook to flush its cache.  Now you go and delete
a block of text in chunk 3, mode A is not warned about it and fails to
flush its cache which now holds completely broken data about chunk 3.

Of course, running the after-change-functions only in the base buffer
won't directly help with that.  I find such discussions depressing,
because they always go back to "indirect buffers are an attractive
nuisance", introducing more problems than they solve.

> The idea of indirect buffers is great.

On its own, it's almost great.  But once you throw in text-properties,
overlays, and buffer-local variables, plus the fact that the choice
between text-properties and overlays is never based on whether "it
should be shared between indirect buffers or not", plus ...  it's just
a big mess that just shuffles problems around without helping
solve them.


        Stefan

Re: trunk r116426: * lisp/jit-lock.el (jit-lock-mode): Keep it disabled in indirect buffers.

[Vitalie Spinu]

 >>> Stefan Monnier on Fri, 23 May 2014 23:10:43 -0400 wrote:

[...]

 > Why prohibit the hooks from the base buffer?  If they're there it's
 > presumably because they're needed.

I thought that foreign chunks should be completely invisible. Your
example disproves the point, indeed. Thanks.

[...]

 > OK, let's say you chunks

 >     chunk 1 in mode A
 >     chunk 2 in mode B
 >     chunk 3 in mode A

 > And let's say that mode A does some clever caching, so it sets up an
 > after-change-functions hook to flush its cache.  Now you go and delete
 > a block of text in chunk 3, mode A is not warned about it and fails to
 > flush its cache which now holds completely broken data about chunk 3.

 > Of course, running the after-change-functions only in the base buffer
 > won't directly help with that.  

If the patch that you propose is ever implemented, local hooks will
still be run in current buffer, right?


Coming back to the original jit-lock problem. I still have no clue how
to fix it without copying the whole initialization part of jit-lock into
polymode.

 > So when jit-lock is triggered it has to refontify in all buffers that
 > share the same base buffer.  And if font-lock is activated in several
 > buffers which share the same base buffer, they'll fight over their
 > shared `face' text-property.

I have just tried this in emacs 24.3.1 and font-lock is triggered only
once in indirect buffers. So it's either some corner case that you are
talking about, or font-lock was broken after 24.3.1 and then you "fixed"
it in:

   commit 6e5d23387db48e9bdf116e59bf7ef717c61982a3
   Author: Stefan Monnier <monnier@iro.umontreal.ca>
   Date:   Wed Feb 12 22:29:47 2014 -0500
   
       * lisp/jit-lock.el (jit-lock-mode): Keep it disabled in indirect buffers.


This patch doesn't affect cloning and jit-lock is still active in cloned
indirect buffers.

   
Vitalie

Re: trunk r116426: * lisp/jit-lock.el (jit-lock-mode): Keep it disabled in indirect buffers.

[Stefan Monnier]

> If the patch that you propose is ever implemented, local hooks will
> still be run in current buffer, right?

No, my intention is to only run the hooks in the base buffer.  It would
then be the responsability of some other package (e.g. polymode) to add
a function to the base-buffer hook to also run the hook in those
indirect buffers that need it.

> Coming back to the original jit-lock problem. I still have no clue how
> to fix it without copying the whole initialization part of jit-lock into
> polymode.

Enable font-lock in the base buffer (and only there) using polymode's
font-lock rules (which will dispatch to the appropriate indirect buffer).

>> So when jit-lock is triggered it has to refontify in all buffers that
>> share the same base buffer.  And if font-lock is activated in several
>> buffers which share the same base buffer, they'll fight over their
>> shared `face' text-property.
> I have just tried this in emacs 24.3.1 and font-lock is triggered only
> once in indirect buffers.

Yes: "has to" is a requirement that was never fulfilled.

> This patch doesn't affect cloning and jit-lock is still active in cloned
> indirect buffers.

Right, that's arguably a bug as well.  Not sure how important it is to
fix it.


        Stefan

Re: trunk r116426: * lisp/jit-lock.el (jit-lock-mode): Keep it disabled in indirect buffers.

[Vitalie Spinu]

 >>> Stefan Monnier on Sat, 24 May 2014 10:35:11 -0400 wrote:

 >> If the patch that you propose is ever implemented, local hooks will
 >> still be run in current buffer, right?

 > No, my intention is to only run the hooks in the base buffer.  

How about running the hooks in all buffers but defining a local version
of the hook as a list with explicit indicator of where to run the hook?
For example (xxx :current), (xxx :base), (xxx :indirect-when-current).

Then jit-lock-after-change could be declared as (jit-lock-after-change :base).

[...]

 >> Coming back to the original jit-lock problem. I still have no clue how
 >> to fix it without copying the whole initialization part of jit-lock into
 >> polymode.

 > Enable font-lock in the base buffer (and only there) using polymode's
 > font-lock rules (which will dispatch to the appropriate indirect buffer).

This would work if your idea is implemented, e.g. change-functions are
executed in the base buffer. Currently, modifications of indirect
buffers don't trigger font-lock at all.

I have activated font-lock in the indirect buffers by copy-pasting the
jit-lock-mode code. It works, but I am noticing occasional emacs crashes
on backward-delete-char-untabify and kill-word which don't occur in
emacs 24.3.

  Vitalie

Re: trunk r116426: * lisp/jit-lock.el (jit-lock-mode): Keep it disabled in indirect buffers.

[Stefan Monnier]

>> No, my intention is to only run the hooks in the base buffer.  
> How about running the hooks in all buffers but defining a local
> version of the hook as a list with explicit indicator of where to run
> the hook?  For example (xxx :current), (xxx :base),
> (xxx :indirect-when-current).
> Then jit-lock-after-change could be declared as
> (jit-lock-after-change :base).

I don't think it's worth the trouble when the Elisp code can do it
all itself.

> This would work if your idea is implemented, e.g. change-functions are
> executed in the base buffer.

Yes, obviously, that's an indispensable step.

> It works, but I am noticing occasional emacs crashes on
> backward-delete-char-untabify and kill-word which don't occur in emacs
> 24.3.

Emacs crashes are bugs that should be reported.


        Stefan