Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Dmitry Gutov]

On 09.11.2019 16:40, Alan Mackenzie wrote:
> +    (widen)

Could you try and honor font-lock-dont-widen here?

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Alan Mackenzie]

Hello, Dmitry.

On Mon, Nov 11, 2019 at 18:52:34 +0200, Dmitry Gutov wrote:
> On 09.11.2019 16:40, Alan Mackenzie wrote:
> > +    (widen)

> Could you try and honor font-lock-dont-widen here?

Difficult.  In CC Mode it is sometimes necessary to enlarge a font lock
region so as to get the context needed to parse some element of the
buffer.  This was what gave rise to the bug.

In all of Emacs, there's just one use of font-lock-dont-widen, in
rmail.el, and that is almost certainly for a different reason than your
use of it (which is, I believe, having several major modes in a single
buffer).

Could you, perhaps, suggest some alternative to using widen in
c-font-lock-fontify-region?  Some way which would still work for CC Mode
(including the just fixed bug), but wouldn't adversely affect the
several major mode code you're maintaining?  Would it be possible to
widen "a bit" rather than widening to the whole buffer?  If so, how
would I determine the bit to widen to?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Dmitry Gutov]

Hi Alan,

On 11.11.2019 22:34, Alan Mackenzie wrote:

> On Mon, Nov 11, 2019 at 18:52:34 +0200, Dmitry Gutov wrote:
>> On 09.11.2019 16:40, Alan Mackenzie wrote:
>>> +    (widen)
> 
>> Could you try and honor font-lock-dont-widen here?
> 
> Difficult.  In CC Mode it is sometimes necessary to enlarge a font lock
> region so as to get the context needed to parse some element of the
> buffer.  This was what gave rise to the bug.

Did that really happen in the reported scenario? I mean, 
font-lock-dont-widen is usually nil, so font-lock will call widen for 
you already.

> In all of Emacs, there's just one use of font-lock-dont-widen, in
> rmail.el, and that is almost certainly for a different reason than your
> use of it (which is, I believe, having several major modes in a single
> buffer).

Does RMail use it for quoted pirces of code? If so, it would be similar.

> Could you, perhaps, suggest some alternative to using widen in
> c-font-lock-fontify-region?  Some way which would still work for CC Mode
> (including the just fixed bug), but wouldn't adversely affect the
> several major mode code you're maintaining?  Would it be possible to
> widen "a bit" rather than widening to the whole buffer?  If so, how
> would I determine the bit to widen to?

Not sure what difference the supposed "bit" would make (and thus, how 
would mmm-mode determine it). But I think that would depend on your 
answer to my first question in this email. Because, I think, normally 
you don't "have to" widen at all, font-lock will do that for you.

In mmm-mode context, however, we apply definite boundaries to the region 
chunks. Here's an example of some Noweb code: 
https://en.wikipedia.org/wiki/Noweb#Example_of_a_simple_noweb_program

The inside of hello.c block would be narrowed to.

Now, I have remembered that CC Mode calls widen from many places 
already, so it already is problematic for using in a context like that. 
Thus this particular change doesn't change the picture much, and so 
please feel free to drop this conversation at any point of time.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Alan Mackenzie]

Hello Dmitry.

On Tue, Nov 12, 2019 at 15:36:08 +0200, Dmitry Gutov wrote:
> Hi Alan,

> On 11.11.2019 22:34, Alan Mackenzie wrote:

> > On Mon, Nov 11, 2019 at 18:52:34 +0200, Dmitry Gutov wrote:
> >> On 09.11.2019 16:40, Alan Mackenzie wrote:
> >>> +    (widen)

> >> Could you try and honor font-lock-dont-widen here?

> > Difficult.  In CC Mode it is sometimes necessary to enlarge a font
> > lock region so as to get the context needed to parse some element of
> > the buffer.  This was what gave rise to the bug.

> Did that really happen in the reported scenario? I mean,
> font-lock-dont-widen is usually nil, so font-lock will call widen for
> you already.

Cutting and pasting from the bug thread for bug #38049:

    [ Juri Linkov ]:
    > This is a reproducible test case:

    > 0. emacs -Q
    > 1. C-x C-f emacs/lib-src/emacsclient.c
    > 2. M-: (progn (search-forward "create-frame" nil t)
    > (reposition-window))

    > Then half screen displays unfontified lines.

    > Fontification doesn't fail in other modes, only in C mode.

    > This has something to do with interaction between c-font-lock
    > and buffer navigation in reposition-window.

    [ Alan Mackenzie ]:
    Indeed it does.

    Briefly,
    (i) reposition-window narrows to (2758 3940) in
      repos-count-screen-lines.
    (ii) This latter function uses vertical-motion to count the lines.
    (iii) vertical-motion triggers jit-lock fontification.
    (iv) This calls (eventually) c-font-lock-fontify-region.
    (v) c-font-lock-fontify-region attempts to examine buffer text before
      the start of the jit-lock chunk to find syntactic context.
    (vi) This is outside the visible region, so Emacs raises an exception.
    (vii) The exception is caught and discarded by an unwind-protect in
      c-font-lock-fontify-region.
    (viii) The jit-lock chunk remains unfontified.


> > In all of Emacs, there's just one use of font-lock-dont-widen, in
> > rmail.el, and that is almost certainly for a different reason than
> > your use of it (which is, I believe, having several major modes in a
> > single buffer).

> Does RMail use it for quoted pirces of code? If so, it would be
> similar.

I don't know RMail at all.  I imagine font-lock-dont-widen is used with
the region as the text of a mail message, excluding things like the
headers.  But I don't know.

> > Could you, perhaps, suggest some alternative to using widen in
> > c-font-lock-fontify-region?  Some way which would still work for CC
> > Mode (including the just fixed bug), but wouldn't adversely affect
> > the several major mode code you're maintaining?  Would it be
> > possible to widen "a bit" rather than widening to the whole buffer?
> > If so, how would I determine the bit to widen to?

> Not sure what difference the supposed "bit" would make (and thus, how 
> would mmm-mode determine it).

I was thinking about, perhaps, widening no wider than the CC Mode
portion of the several major mode buffer.

> But I think that would depend on your answer to my first question in
> this email. Because, I think, normally you don't "have to" widen at
> all, font-lock will do that for you.

I think it does, normally, but it didn't when the font locking was
invoked for vertical-motion.  (Font locking is needed to determine the
faces used, which might have different vertical sizes.)

> In mmm-mode context, however, we apply definite boundaries to the
> region chunks. Here's an example of some Noweb code:
> https://en.wikipedia.org/wiki/Noweb#Example_of_a_simple_noweb_program

> The inside of hello.c block would be narrowed to.

I think I've said this before, but I don't think narrowing is the right
tool for that task.  I don't think there is a suitable tool in Emacs at
the moment.

> Now, I have remembered that CC Mode calls widen from many places 
> already, so it already is problematic for using in a context like that. 

It does, yes.  Users also use widening and narrowing.

> Thus this particular change doesn't change the picture much, and so 
> please feel free to drop this conversation at any point of time.

OK.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Dmitry Gutov]

Hi Alan,

On 13.11.2019 23:19, Alan Mackenzie wrote:

>> Did that really happen in the reported scenario? I mean,
>> font-lock-dont-widen is usually nil, so font-lock will call widen for
>> you already.
> 
> Cutting and pasting from the bug thread for bug #38049:
> 
>      [ Juri Linkov ]:
>      > This is a reproducible test case:
> 
>      > 0. emacs -Q
>      > 1. C-x C-f emacs/lib-src/emacsclient.c
>      > 2. M-: (progn (search-forward "create-frame" nil t)
>      > (reposition-window))
> 
>      > Then half screen displays unfontified lines.
> 
>      > Fontification doesn't fail in other modes, only in C mode.
> 
>      > This has something to do with interaction between c-font-lock
>      > and buffer navigation in reposition-window.
> 
>      [ Alan Mackenzie ]:
>      Indeed it does.
> 
>      Briefly,
>      (i) reposition-window narrows to (2758 3940) in
>        repos-count-screen-lines.
>      (ii) This latter function uses vertical-motion to count the lines.
>      (iii) vertical-motion triggers jit-lock fontification.
>      (iv) This calls (eventually) c-font-lock-fontify-region.
>      (v) c-font-lock-fontify-region attempts to examine buffer text before
>        the start of the jit-lock chunk to find syntactic context.
>      (vi) This is outside the visible region, so Emacs raises an exception.
>      (vii) The exception is caught and discarded by an unwind-protect in
>        c-font-lock-fontify-region.
>      (viii) The jit-lock chunk remains unfontified.

Very interesting, thank you. I wonder if the patch below would fix it as 
well. Though, if course, that can't account for all possible uses of 
vertical-motion inside a narrowed buffer.

diff --git a/lisp/reposition.el b/lisp/reposition.el
index 22f9986efb..131e595858 100644
--- a/lisp/reposition.el
+++ b/lisp/reposition.el
@@ -172,6 +172,8 @@ reposition-window

  ;; Return number of screen lines between START and END.
  (defun repos-count-screen-lines (start end)
+  ;; Make sure jit-lock doesn't have to run in a narrowed context.
+  (font-lock-ensure start end)
    (save-excursion
      (save-restriction
        (narrow-to-region start end)


>>> Could you, perhaps, suggest some alternative to using widen in
>>> c-font-lock-fontify-region?  Some way which would still work for CC
>>> Mode (including the just fixed bug), but wouldn't adversely affect
>>> the several major mode code you're maintaining?  Would it be
>>> possible to widen "a bit" rather than widening to the whole buffer?
>>> If so, how would I determine the bit to widen to?
> 
>> Not sure what difference the supposed "bit" would make (and thus, how
>> would mmm-mode determine it).
> 
> I was thinking about, perhaps, widening no wider than the CC Mode
> portion of the several major mode buffer.

That would normally be what the buffer is narrowed to already. Though I 
suppose we could also offer a function that would re-narrow to these 
bounds on demand.

>> But I think that would depend on your answer to my first question in
>> this email. Because, I think, normally you don't "have to" widen at
>> all, font-lock will do that for you.
> 
> I think it does, normally, but it didn't when the font locking was
> invoked for vertical-motion.  (Font locking is needed to determine the
> faces used, which might have different vertical sizes.)

Yes. I never thought about this before.

>> In mmm-mode context, however, we apply definite boundaries to the
>> region chunks. Here's an example of some Noweb code:
>> https://en.wikipedia.org/wiki/Noweb#Example_of_a_simple_noweb_program
> 
>> The inside of hello.c block would be narrowed to.
> 
> I think I've said this before, but I don't think narrowing is the right
> tool for that task.  I don't think there is a suitable tool in Emacs at
> the moment.

*shrug* We do the best with what we have.

>> Now, I have remembered that CC Mode calls widen from many places
>> already, so it already is problematic for using in a context like that.
> 
> It does, yes.  Users also use widening and narrowing.

It's a different thing. Strategically undoing interactively applied 
narrowing is not hard.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 12 Nov 2019 15:36:08 +0200
> Cc: emacs-devel@gnu.org
> 
> > In all of Emacs, there's just one use of font-lock-dont-widen, in
> > rmail.el, and that is almost certainly for a different reason than your
> > use of it (which is, I believe, having several major modes in a single
> > buffer).
> 
> Does RMail use it for quoted pirces of code? If so, it would be similar.

No, Rmail uses that variable because each email message it displays is
just a small portion of a large mbox file, and Rmail narrows the
buffer to exclude all the other email messages.  So it wants font-lock
to work only on the narrowed portion of the buffer, as that presents a
single logical unit -- one email message -- to be fontified.  So it
has nothing to do with fragments of code embedded in text.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Eli Zaretskii]

> Date: Wed, 13 Nov 2019 21:19:36 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: emacs-devel@gnu.org
> 
> > > Difficult.  In CC Mode it is sometimes necessary to enlarge a font
> > > lock region so as to get the context needed to parse some element of
> > > the buffer.  This was what gave rise to the bug.
> 
> > Did that really happen in the reported scenario? I mean,
> > font-lock-dont-widen is usually nil, so font-lock will call widen for
> > you already.
> 
> Cutting and pasting from the bug thread for bug #38049:

But that couldn't happen in the use case with code fragments, could
it?  The mode which manages such buffers narrows the buffer to include
the entire code fragment, so no reason for CC mode to widen.  Right?

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Stefan Monnier]

>  ;; Return number of screen lines between START and END.
>  (defun repos-count-screen-lines (start end)
> +  ;; Make sure jit-lock doesn't have to run in a narrowed context.
> +  (font-lock-ensure start end)
>    (save-excursion
>      (save-restriction
>        (narrow-to-region start end)

Hmm... the buffer might have font-lock disabled, in which case we'd
be fontifying where the user didn't want fontification.

But I haven't followed the thread enough to know what we're talking
about, really.  E.g. I have no idea what's the connection between
CC-mode, font-lock-dont-widen, and the above hunk.  More specifically,
when looking at the above patch my first reaction is "if it's needed
here, why isn't it needed almost everywhere else as well?"


        Stefan

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 14 Nov 2019 00:33:33 +0200
> Cc: emacs-devel@gnu.org
> 
> Very interesting, thank you. I wonder if the patch below would fix it as 
> well. Though, if course, that can't account for all possible uses of 
> vertical-motion inside a narrowed buffer.
> 
> diff --git a/lisp/reposition.el b/lisp/reposition.el
> index 22f9986efb..131e595858 100644
> --- a/lisp/reposition.el
> +++ b/lisp/reposition.el
> @@ -172,6 +172,8 @@ reposition-window
> 
>   ;; Return number of screen lines between START and END.
>   (defun repos-count-screen-lines (start end)
> +  ;; Make sure jit-lock doesn't have to run in a narrowed context.
> +  (font-lock-ensure start end)

Isn't this a kludge of sorts?  We want CC mode not to fail to fontify
if it's called in a narrowed context, regardless of how that happened,
and you seem to be fixing just this particular situation?  The actual
problem is much more general, AFAIU.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Dmitry Gutov]

On 14.11.2019 15:42, Stefan Monnier wrote:
>>   ;; Return number of screen lines between START and END.
>>   (defun repos-count-screen-lines (start end)
>> +  ;; Make sure jit-lock doesn't have to run in a narrowed context.
>> +  (font-lock-ensure start end)
>>     (save-excursion
>>       (save-restriction
>>         (narrow-to-region start end)
> 
> Hmm... the buffer might have font-lock disabled, in which case we'd
> be fontifying where the user didn't want fontification.

We could wrap it in (when font-lock-mode ...), then? Or something to 
that effect.

> But I haven't followed the thread enough to know what we're talking
> about, really.  E.g. I have no idea what's the connection between
> CC-mode, font-lock-dont-widen, and the above hunk.  More specifically,
> when looking at the above patch my first reaction is "if it's needed
> here, why isn't it needed almost everywhere else as well?"

The hope is that calling vertical-motion (or font-lock-ensure directly) 
inside narrowing is a rare occasion, and it doesn't happen in a lot of 
functions. At least that has been my experience.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Dmitry Gutov]

On 14.11.2019 15:49, Eli Zaretskii wrote:
> Isn't this a kludge of sorts?  We want CC mode not to fail to fontify
> if it's called in a narrowed context, regardless of how that happened,
> and you seem to be fixing just this particular situation?  The actual
> problem is much more general, AFAIU.

Well, normally it wouldn't be called in a narrowed context (because 
font-lock already widens by default). Except when inside a multiple-mode 
context.

But there are tricky scenarios where it happens unexpectedly. So we'll 
have to take care of them somehow.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 14 Nov 2019 16:05:13 +0200
> Cc: Alan Mackenzie <acm@muc.de>, emacs-devel@gnu.org
> 
> The hope is that calling vertical-motion (or font-lock-ensure directly) 
> inside narrowing is a rare occasion

Why do we need to have such hopes?  The reality could be different,
and I'd like us to support those different conditions.  It doesn't
sound like it's much harder.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Dmitry Gutov]

On 14.11.2019 16:29, Eli Zaretskii wrote:
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Thu, 14 Nov 2019 16:05:13 +0200
>> Cc: Alan Mackenzie <acm@muc.de>, emacs-devel@gnu.org
>>
>> The hope is that calling vertical-motion (or font-lock-ensure directly)
>> inside narrowing is a rare occasion
> 
> Why do we need to have such hopes?

For starters, because it's the current expectation of mmm-mode. :-)

That's selfish, I know. But the counter-examples are sufficiently rare.

> The reality could be different,
> and I'd like us to support those different conditions.  It doesn't
> sound like it's much harder.

There are options, sure. But I think all of them will require explicit 
support from major modes. Like adding a new element to 
prog-indentation-context, which will be a function which either returns 
new narrowing bounds or does the narrowing itself. And then the major 
mode will have to make sure to use it everywhere.

This particular case is annoying because it seems to mean that the 
(widen) call in font-lock-default-fontify-buffer doesn't help, and a lot 
of font-lock-keywords matchers will need to call (widen) anyway.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Dmitry Gutov]

On 14.11.2019 0:33, Dmitry Gutov wrote:
> 
> Very interesting, thank you. I wonder if the patch below would fix it as 
> well.

BTW, here's an alternative patch. If CC Mode is fine with only seeing 
the beginning of the buffer until point (for the purpose of maintaining 
its caches), this should work just as well, and it won't have to add a 
(widen) call itself.

diff --git a/lisp/reposition.el b/lisp/reposition.el
index 22f9986efb..5d9c3180ad 100644
--- a/lisp/reposition.el
+++ b/lisp/reposition.el
@@ -174,8 +174,8 @@ reposition-window
  (defun repos-count-screen-lines (start end)
    (save-excursion
      (save-restriction
-      (narrow-to-region start end)
-      (goto-char (point-min))
+      (narrow-to-region (point-min) end)
+      (goto-char start)
        (vertical-motion (- (point-max) (point-min))))))

  ;; Return number of screen lines between START and END; returns a negative

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 14 Nov 2019 16:35:14 +0200
> 
> On 14.11.2019 16:29, Eli Zaretskii wrote:
> >> From: Dmitry Gutov <dgutov@yandex.ru>
> >> Date: Thu, 14 Nov 2019 16:05:13 +0200
> >> Cc: Alan Mackenzie <acm@muc.de>, emacs-devel@gnu.org
> >>
> >> The hope is that calling vertical-motion (or font-lock-ensure directly)
> >> inside narrowing is a rare occasion
> > 
> > Why do we need to have such hopes?
> 
> For starters, because it's the current expectation of mmm-mode. :-)

It's an unrealistic expectation.  Code that narrows a buffer and then
invokes something that calls vertical-motion is not special in any
case, and could happen anywhere.

> > The reality could be different,
> > and I'd like us to support those different conditions.  It doesn't
> > sound like it's much harder.
> 
> There are options, sure. But I think all of them will require explicit 
> support from major modes.

I don't think so.  All we need to make sure is that the narrowing
doesn't conceal portions of the embedded fragment.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Dmitry Gutov]

On 14.11.2019 16:48, Eli Zaretskii wrote:
>>> The reality could be different,
>>> and I'd like us to support those different conditions.  It doesn't
>>> sound like it's much harder.
>> There are options, sure. But I think all of them will require explicit
>> support from major modes.
> I don't think so.  All we need to make sure is that the narrowing
> doesn't conceal portions of the embedded fragment.

I would very much like to read your idea in more detail.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Stefan Monnier]

> diff --git a/lisp/reposition.el b/lisp/reposition.el
> index 22f9986efb..5d9c3180ad 100644
> --- a/lisp/reposition.el
> +++ b/lisp/reposition.el
> @@ -174,8 +174,8 @@ reposition-window
>  (defun repos-count-screen-lines (start end)
>    (save-excursion
>      (save-restriction
> -      (narrow-to-region start end)
> -      (goto-char (point-min))
> +      (narrow-to-region (point-min) end)
> +      (goto-char start)
>        (vertical-motion (- (point-max) (point-min))))))

This looks like a good change in any case, tho it does beg the question:
why does it narrow?

Also, I think I'm beginning to understand: the issue is that
vertical-motion uses the redisplay code, which in turn calls jit-lock
when needed.  So, your earlier patch which added a call to
`font-lock-ensure` was "more or less right", except that it should have
used `jit-lock-ensure` (currently called `jit-lock-fontify-now`) and
could have a comment explaining that vertical-motion will trigger
jit-lock anyway so it's better to do it once beforehand on the whole
region, not only so it can be done outside of the narrowing but also so
it can be done more efficiently than one jit-lock-chunk-size at a time.


        Stefan

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Dmitry Gutov]

On 14.11.2019 16:55, Stefan Monnier wrote:
>> diff --git a/lisp/reposition.el b/lisp/reposition.el
>> index 22f9986efb..5d9c3180ad 100644
>> --- a/lisp/reposition.el
>> +++ b/lisp/reposition.el
>> @@ -174,8 +174,8 @@ reposition-window
>>   (defun repos-count-screen-lines (start end)
>>     (save-excursion
>>       (save-restriction
>> -      (narrow-to-region start end)
>> -      (goto-char (point-min))
>> +      (narrow-to-region (point-min) end)
>> +      (goto-char start)
>>         (vertical-motion (- (point-max) (point-min))))))
> 
> This looks like a good change in any case, tho it does beg the question:
> why does it narrow?

I think it does that simply because vertical-motion takes LINES as an 
argument, and not LIMIT. And it's easier to narrow the buffer than 
compute the number of lines between start and end.

> Also, I think I'm beginning to understand: the issue is that
> vertical-motion uses the redisplay code, which in turn calls jit-lock
> when needed.

Exactly.

> So, your earlier patch which added a call to
> `font-lock-ensure` was "more or less right", except that it should have
> used `jit-lock-ensure` (currently called `jit-lock-fontify-now`)

But wouldn't this fail to work when jit-lock is not used? And we'll need 
a guard to make sure font-lock is used as well. (*)

> and
> could have a comment explaining that vertical-motion will trigger
> jit-lock anyway so it's better to do it once beforehand on the whole
> region, not only so it can be done outside of the narrowing but also so
> it can be done more efficiently than one jit-lock-chunk-size at a time.

That's a good explanation as well, yes.

So, if you like, I can push a patch with both of these changes. If you 
could only clarify what condition to use in (*).

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Stefan Monnier]

>>> diff --git a/lisp/reposition.el b/lisp/reposition.el
>>> index 22f9986efb..5d9c3180ad 100644
>>> --- a/lisp/reposition.el
>>> +++ b/lisp/reposition.el
>>> @@ -174,8 +174,8 @@ reposition-window
>>>   (defun repos-count-screen-lines (start end)
>>>     (save-excursion
>>>       (save-restriction
>>> -      (narrow-to-region start end)
>>> -      (goto-char (point-min))
>>> +      (narrow-to-region (point-min) end)
>>> +      (goto-char start)
>>>         (vertical-motion (- (point-max) (point-min))))))
>> This looks like a good change in any case, tho it does beg the question:
>> why does it narrow?
>
> I think it does that simply because vertical-motion takes LINES as an
> argument, and not LIMIT. And it's easier to narrow the buffer than 
> compute the number of lines between start and end.

Ah, right, that makes total sense.  So, a big +1 on this patch from me.

>> So, your earlier patch which added a call to `font-lock-ensure` was
>> "more or less right", except that it should have used
>> `jit-lock-ensure` (currently called `jit-lock-fontify-now`)
> But wouldn't this fail to work when jit-lock is not used?

I don't think so, no: it will (should) just do nothing.
But in case it misbehaves, I'd consider it a bug in jit-lock.el.

> And we'll need a guard to make sure font-lock is used as well. (*)

Why would we need that?


        Stefan

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 14 Nov 2019 16:50:14 +0200
> 
> On 14.11.2019 16:48, Eli Zaretskii wrote:
> >>> The reality could be different,
> >>> and I'd like us to support those different conditions.  It doesn't
> >>> sound like it's much harder.
> >> There are options, sure. But I think all of them will require explicit
> >> support from major modes.
> > I don't think so.  All we need to make sure is that the narrowing
> > doesn't conceal portions of the embedded fragment.
> 
> I would very much like to read your idea in more detail.

I'm not sure what else is there to say.  Really.  Can you ask specific
questions?

My point was that Alan wanted not to start fontification from a random
place determined by blindly narrowing to some arbitrary chunk.  But in
mmm-mode the code fragment is present in the narrowed portion in its
entirety, so it doesn't sound like we have a real problem here.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 14 Nov 2019 09:55:58 -0500
> Cc: Alan Mackenzie <acm@muc.de>, emacs-devel@gnu.org
> 
> Also, I think I'm beginning to understand: the issue is that
> vertical-motion uses the redisplay code, which in turn calls jit-lock
> when needed.

Yes.

> So, your earlier patch which added a call to `font-lock-ensure` was
> "more or less right", except that it should have used
> `jit-lock-ensure` (currently called `jit-lock-fontify-now`) and
> could have a comment explaining that vertical-motion will trigger
> jit-lock anyway so it's better to do it once beforehand on the whole
> region, not only so it can be done outside of the narrowing but also
> so it can be done more efficiently than one jit-lock-chunk-size at a
> time.

I'd like to solve this in CC mode instead, because the problem is
there.  Fontifying arbitrary portions of the buffer to indirectly
avoid triggering the problem in CC mode sounds not TRT to me.  In
particular, what if the POINT-MIN..END chunk is still too large to
fontify in one go?

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Stefan Monnier]

>> So, your earlier patch which added a call to `font-lock-ensure` was
>> "more or less right", except that it should have used
>> `jit-lock-ensure` (currently called `jit-lock-fontify-now`) and
>> could have a comment explaining that vertical-motion will trigger
>> jit-lock anyway so it's better to do it once beforehand on the whole
>> region, not only so it can be done outside of the narrowing but also
>> so it can be done more efficiently than one jit-lock-chunk-size at a
>> time.
> I'd like to solve this in CC mode instead, because the problem is there.

I think the problem is wider than CC-mode.  Maybe CC-mode is more
susceptible to it or maybe it's just an accident that this was reported
for CC-mode, but running font-lock (and syntax-propertize) within
narrowing tends to be fiddly.

> Fontifying arbitrary portions of the buffer to indirectly
> avoid triggering the problem in CC mode sounds not TRT to me.

`vertical-motion` *will* call `jit-lock-fontify-now` on all those buffer
positions in any case, so while it is extra code I'd rather we don't need
to write, it shouldn't have a negative impact on performance, quite the
contrary.

> In particular, what if the POINT-MIN..END chunk is still too large to
> fontify in one go?

AFAIK all uses of jit-lock are more efficient if they get fewer larger
chunks than more smaller chunks.


        Stefan

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  acm@muc.de,  emacs-devel@gnu.org
> Date: Thu, 14 Nov 2019 13:51:05 -0500
> 
> >> So, your earlier patch which added a call to `font-lock-ensure` was
> >> "more or less right", except that it should have used
> >> `jit-lock-ensure` (currently called `jit-lock-fontify-now`) and
> >> could have a comment explaining that vertical-motion will trigger
> >> jit-lock anyway so it's better to do it once beforehand on the whole
> >> region, not only so it can be done outside of the narrowing but also
> >> so it can be done more efficiently than one jit-lock-chunk-size at a
> >> time.
> > I'd like to solve this in CC mode instead, because the problem is there.
> 
> I think the problem is wider than CC-mode.  Maybe CC-mode is more
> susceptible to it or maybe it's just an accident that this was reported
> for CC-mode, but running font-lock (and syntax-propertize) within
> narrowing tends to be fiddly.

I'm also okay with fixing it in font-lock or jit-lock.  But doing that
inside reposition.el makes no sense to me.

> > Fontifying arbitrary portions of the buffer to indirectly
> > avoid triggering the problem in CC mode sounds not TRT to me.
> 
> `vertical-motion` *will* call `jit-lock-fontify-now` on all those buffer
> positions in any case, so while it is extra code I'd rather we don't need
> to write, it shouldn't have a negative impact on performance, quite the
> contrary.

So we are going to do such changes in every application that calls
vertical-motion, directly or indirectly?  And what about
posn-at-point, or pos-visible-in-window-p, or any other API that uses
the display code internally? are we going to fix their callers as
well?  That really makes no sense, so I very much hope I somehow
grossly misunderstand what you are suggesting as a fix.

> > In particular, what if the POINT-MIN..END chunk is still too large to
> > fontify in one go?
> 
> AFAIK all uses of jit-lock are more efficient if they get fewer larger
> chunks than more smaller chunks.

So you are saying that we should enlarge jit-lock-chunk-size to
most-positive-fixnum?  I don't think so.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Dmitry Gutov]

On 14.11.2019 21:19, Eli Zaretskii wrote:
> So we are going to do such changes in every application that calls
> vertical-motion, directly or indirectly?  And what about
> posn-at-point, or pos-visible-in-window-p, or any other API that uses
> the display code internally? are we going to fix their callers as
> well?  That really makes no sense, so I very much hope I somehow
> grossly misunderstand what you are suggesting as a fix.

Your objections are on point, of course.

It's not a comprehensive fix, but it should be an improvement anyway (at 
least for performance).

> So you are saying that we should enlarge jit-lock-chunk-size to
> most-positive-fixnum?  I don't think so.

No, it's more about the number of calls to font-lock functions. I.e. 
fontifying line-by-line vs. doing it in a larger block. The size of that 
block is not so important, as long as it's much bigger than a line.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Eli Zaretskii]

> Cc: acm@muc.de, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 14 Nov 2019 21:48:24 +0200
> 
> > So you are saying that we should enlarge jit-lock-chunk-size to
> > most-positive-fixnum?  I don't think so.
> 
> No, it's more about the number of calls to font-lock functions. I.e. 
> fontifying line-by-line vs. doing it in a larger block. The size of that 
> block is not so important, as long as it's much bigger than a line.

I agree, of course, and the default value of jit-lock-chunk-size
agrees as well.

But calling jit-lock-ensure as shown might try to fontify much more,
right?

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Dmitry Gutov]

On 14.11.2019 16:55, Stefan Monnier wrote:
> So, your earlier patch which added a call to
> `font-lock-ensure` was "more or less right", except that it should have
> used `jit-lock-ensure` (currently called `jit-lock-fontify-now`)

Stefan, could you also clarify that?

font-lock-ensure-function is set to jit-lock-fontify-now when jit-lock 
is in use. And when it's not, wouldn't it be better to fontify the 
region the "old" way instead of leaving it unfontified?

Is that because font-lock-ensure calls font-lock-set-defaults?

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Dmitry Gutov]

On 14.11.2019 21:56, Eli Zaretskii wrote:
> But calling jit-lock-ensure as shown might try to fontify much more,
> right?

Not sure I understand. Why would it? It's passed the same region bounds.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Dmitry Gutov]

On 14.11.2019 18:08, Eli Zaretskii wrote:
>> Cc: monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Thu, 14 Nov 2019 16:50:14 +0200
>>
>> On 14.11.2019 16:48, Eli Zaretskii wrote:
>>>>> The reality could be different,
>>>>> and I'd like us to support those different conditions.  It doesn't
>>>>> sound like it's much harder.
>>>> There are options, sure. But I think all of them will require explicit
>>>> support from major modes.
>>> I don't think so.  All we need to make sure is that the narrowing
>>> doesn't conceal portions of the embedded fragment.
>>
>> I would very much like to read your idea in more detail.
> 
> I'm not sure what else is there to say.  Really.  Can you ask specific
> questions?

Question one is: do you still stand by that earlier assessment?

> My point was that Alan wanted not to start fontification from a random
> place determined by blindly narrowing to some arbitrary chunk.  But in
> mmm-mode the code fragment is present in the narrowed portion in its
> entirety, so it doesn't sound like we have a real problem here.

It's what I was hoping would work, but apparently font-lock rules are 
inevitably called from different narrowings (at least in some, edge-case 
scenarios). So to be ironclad, CC Mode has to call 'widen' its some of 
its font-lock-keywords matchers.

But that breaks the narrowing applied by mmm-mode in its aggregated 
fontification function.

Several Major Modes. [Was: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.]

[Alan Mackenzie]

Hello, Dmitry.

On Thu, Nov 14, 2019 at 00:33:33 +0200, Dmitry Gutov wrote:

[ .... ]

> >> In mmm-mode context, however, we apply definite boundaries to the
> >> region chunks. Here's an example of some Noweb code:
> >> https://en.wikipedia.org/wiki/Noweb#Example_of_a_simple_noweb_program

> >> The inside of hello.c block would be narrowed to.

> > I think I've said this before, but I don't think narrowing is the right
> > tool for that task.  I don't think there is a suitable tool in Emacs at
> > the moment.

> *shrug* We do the best with what we have.

Why can we not formulate something better, an enhancement to the Emacs
core which would support several major modes properly?  I have made
proposals in this area before, but I think they were to grandiose to be
implementable.

What seems to be needed is a way of partioning a buffer into several
sub-buffers (which I have called "islands" in the past), and having a new
type of local variable, one valid in exactly one sub-buffer.  More or
less.

> >> Now, I have remembered that CC Mode calls widen from many places
> >> already, so it already is problematic for using in a context like
> >> that.

> > It does, yes.  Users also use widening and narrowing.

I believe these problems won't go away, and there will always be
conflicts between CC Mode (as it is) and mmm-mode (as it is).

> It's a different thing. Strategically undoing interactively applied 
> narrowing is not hard.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Stefan Monnier]

>> I think the problem is wider than CC-mode.  Maybe CC-mode is more
>> susceptible to it or maybe it's just an accident that this was reported
>> for CC-mode, but running font-lock (and syntax-propertize) within
>> narrowing tends to be fiddly.
> I'm also okay with fixing it in font-lock or jit-lock.  But doing that
> inside reposition.el makes no sense to me.

IIUC this can't be fixed in font-lock or jit-lock.  It can only be fixed
"upstream" (e.g. in repos-count-screen-lines) or "downstream" (in the
particular major mode's rules).

I think neither "upstream" nor "downstream" is satisfactory, but
until/unless we introduce some "structure" on the narrowings (so that
font-lock can know for sure whether and how it can widen) that's what we
have to live with.

> So we are going to do such changes in every application that calls
> vertical-motion, directly or indirectly?

I'd phrase it as "this would need to be done in any narrowing
which changes point-min and internally can trigger jit-lock".

Whether we're going to do it or not I don't know.
I think "don't change point-min" is generally the better fix (which
partly means weaning ourselves from narrowing).

> And what about posn-at-point, or pos-visible-in-window-p, or any other
> API that uses the display code internally?

If these can trigger jit-lock then that applies as well, yes.
`sit-for` as well, BTW.

> are we going to fix their callers as well?

The fix should not be "where we can do vertical-motion (and friends)"
but "where we narrow (and later trigger jit-lock)".  Whether that means
"fix their callers as well" depends on what it is that owns the "their
callers": yes it would be in the functions that call (directly or not)
`vertical-motion` (and friends) but in the functions that call
those functions.

>> > In particular, what if the POINT-MIN..END chunk is still too large to
>> > fontify in one go?
>> AFAIK all uses of jit-lock are more efficient if they get fewer larger
>> chunks than more smaller chunks.
> So you are saying that we should enlarge jit-lock-chunk-size to
> most-positive-fixnum?

That would cause jit-lock being applied not just to "fewer larger
chunks" but also to more total text: when the redisplay calls jit-lock,
we know POS will be displayed and needs to be jit-locked but we don't
how far further we will keep looking, and that what jit-lock-chunk-size
is for (it's supposed to be large enough that the per-chunk overhead
doesn't kill it while being small enough that the amount of text
we needlessly jit-lock isn't too large either).

In the presently discussed patch we know beforehand that all the text
between start and end will be considered and will need to be jit-locked,
which is why we can do it more efficiently in a single explicit call
than what the redisplay code would do otherwise.

In any case, changing the narrow-to-region so it doesn't change
point-min is the right fix.  Adding a call to `jit-lock-fontify-now` is
another but worse way to fix it.

Adding that call should be harmless and could be beneficial for
performance, but I personally wouldn't do it (it can also be harmful
for performance in the case where the text has already been jit-locked
in which case it won't do anything but will still take a bit of time to
do it).


        Stfan

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Dmitry Gutov]

On 14.11.2019 23:29, Stefan Monnier wrote:
> In any case, changing the narrow-to-region so it doesn't change
> point-min is the right fix.  Adding a call to `jit-lock-fontify-now` is
> another but worse way to fix it.

OK, I've pushed the less-controversial patch.

Re: Several Major Modes. [Was: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.]

[Dmitry Gutov]

On 14.11.2019 23:24, Alan Mackenzie wrote:
> Hello, Dmitry.
> 
> On Thu, Nov 14, 2019 at 00:33:33 +0200, Dmitry Gutov wrote:
> 
> [ .... ]
> 
>>>> In mmm-mode context, however, we apply definite boundaries to the
>>>> region chunks. Here's an example of some Noweb code:
>>>> https://en.wikipedia.org/wiki/Noweb#Example_of_a_simple_noweb_program
> 
>>>> The inside of hello.c block would be narrowed to.
> 
>>> I think I've said this before, but I don't think narrowing is the right
>>> tool for that task.  I don't think there is a suitable tool in Emacs at
>>> the moment.
> 
>> *shrug* We do the best with what we have.
> 
> Why can we not formulate something better, an enhancement to the Emacs
> core which would support several major modes properly?  I have made
> proposals in this area before, but I think they were to grandiose to be
> implementable.
> 
> What seems to be needed is a way of partioning a buffer into several
> sub-buffers (which I have called "islands" in the past), and having a new
> type of local variable, one valid in exactly one sub-buffer.  More or
> less.

There are options. We'd have to decide on a suitable model, calling them 
islands or whatever, but I think the first approximation is to either 
make sure narrowing is available for this purpose or, if we absolutely 
can't make it work, add a new element to prog-indentation-context which 
will be a function that would return the bounds of the current chunk.

Regarding "new type of local variable", mmm-mode already tracks 
something like that.

>>>> Now, I have remembered that CC Mode calls widen from many places
>>>> already, so it already is problematic for using in a context like
>>>> that.
> 
>>> It does, yes.  Users also use widening and narrowing.
> 
> I believe these problems won't go away, and there will always be
> conflicts between CC Mode (as it is) and mmm-mode (as it is).

I think we should also try to understand whether making CC Mode play 
nice to doable/feasible, and for what uses. Like, I think it can work 
(more or less) already when it's the primary major mode (meaning the 
buffer starts and ends with it), so the embedded chunks are all other 
modes. Is it feasible to support embedded chunks? To support chunks with 
incomplete pieces of code (which are e.g. included conditionally by the 
surrounding template)? By answering these questions we can temper our 
expectations and come up with a practical plan.

Doing nothing is also a valid choice, BTW, since for many uses replacing 
c-mode with js-mode works pretty okay. I've been recommending it to 
MMM-mode users.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 14 Nov 2019 21:59:50 +0200
> 
> On 14.11.2019 21:56, Eli Zaretskii wrote:
> > But calling jit-lock-ensure as shown might try to fontify much more,
> > right?
> 
> Not sure I understand. Why would it? It's passed the same region bounds.

The suggestion AFAIU was to pass point-min as the start, and that
could be anything.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Dmitry Gutov]

On 15.11.2019 9:34, Eli Zaretskii wrote:
>> Cc: monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Thu, 14 Nov 2019 21:59:50 +0200
>>
>> On 14.11.2019 21:56, Eli Zaretskii wrote:
>>> But calling jit-lock-ensure as shown might try to fontify much more,
>>> right?
>>
>> Not sure I understand. Why would it? It's passed the same region bounds.
> 
> The suggestion AFAIU was to pass point-min as the start, and that
> could be anything.

No, I was passing start as the start. In any case, it would've been a 
bug in the patch, not in the whole idea.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 14 Nov 2019 23:07:53 +0200
> 
> On 14.11.2019 18:08, Eli Zaretskii wrote:
> >> Cc: monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> >> From: Dmitry Gutov <dgutov@yandex.ru>
> >> Date: Thu, 14 Nov 2019 16:50:14 +0200
> >>
> >> On 14.11.2019 16:48, Eli Zaretskii wrote:
> >>>>> The reality could be different,
> >>>>> and I'd like us to support those different conditions.  It doesn't
> >>>>> sound like it's much harder.
> >>>> There are options, sure. But I think all of them will require explicit
> >>>> support from major modes.
> >>> I don't think so.  All we need to make sure is that the narrowing
> >>> doesn't conceal portions of the embedded fragment.
> >>
> >> I would very much like to read your idea in more detail.
> > 
> > I'm not sure what else is there to say.  Really.  Can you ask specific
> > questions?
> 
> Question one is: do you still stand by that earlier assessment?

Which one?

> > My point was that Alan wanted not to start fontification from a random
> > place determined by blindly narrowing to some arbitrary chunk.  But in
> > mmm-mode the code fragment is present in the narrowed portion in its
> > entirety, so it doesn't sound like we have a real problem here.
> 
> It's what I was hoping would work, but apparently font-lock rules are 
> inevitably called from different narrowings (at least in some, edge-case 
> scenarios). So to be ironclad, CC Mode has to call 'widen' its some of 
> its font-lock-keywords matchers.
> 
> But that breaks the narrowing applied by mmm-mode in its aggregated 
> fontification function.

Maybe we need to revisit the prog-widen idea.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  acm@muc.de,  emacs-devel@gnu.org
> Date: Thu, 14 Nov 2019 16:29:38 -0500
> 
> >> I think the problem is wider than CC-mode.  Maybe CC-mode is more
> >> susceptible to it or maybe it's just an accident that this was reported
> >> for CC-mode, but running font-lock (and syntax-propertize) within
> >> narrowing tends to be fiddly.
> > I'm also okay with fixing it in font-lock or jit-lock.  But doing that
> > inside reposition.el makes no sense to me.
> 
> IIUC this can't be fixed in font-lock or jit-lock.

Why not?

> It can only be fixed "upstream" (e.g. in repos-count-screen-lines)
> or "downstream" (in the particular major mode's rules).

I believe my original suggestion was to fix this in CC Mode, but you
seemed to disagree.  So now you do agree with that?

> > So we are going to do such changes in every application that calls
> > vertical-motion, directly or indirectly?
> 
> I'd phrase it as "this would need to be done in any narrowing
> which changes point-min and internally can trigger jit-lock".

That's any number of places out there, and I don't see how could they
be identified.

> The fix should not be "where we can do vertical-motion (and friends)"
> but "where we narrow (and later trigger jit-lock)".  Whether that means
> "fix their callers as well" depends on what it is that owns the "their
> callers": yes it would be in the functions that call (directly or not)
> `vertical-motion` (and friends) but in the functions that call
> those functions.

Sorry, I don't think this is practical.  I think fixing this in CC
Mode is a much better alternative.

> In any case, changing the narrow-to-region so it doesn't change
> point-min is the right fix.  Adding a call to `jit-lock-fontify-now` is
> another but worse way to fix it.

I don't think any of these two ways were proposed as patches, so I
don't think I understand well enough what are those changes you prefer
here.

> Adding that call should be harmless and could be beneficial for
> performance, but I personally wouldn't do it (it can also be harmful
> for performance in the case where the text has already been jit-locked
> in which case it won't do anything but will still take a bit of time to
> do it).

What is "that call" in this context?

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Dmitry Gutov]

On 15.11.2019 11:31, Eli Zaretskii wrote:
>>>>>>> The reality could be different,
>>>>>>> and I'd like us to support those different conditions.  It doesn't
>>>>>>> sound like it's much harder.
>>>>>> There are options, sure. But I think all of them will require explicit
>>>>>> support from major modes.
>>>>> I don't think so.  All we need to make sure is that the narrowing
>>>>> doesn't conceal portions of the embedded fragment.
>>>> I would very much like to read your idea in more detail.
>>> I'm not sure what else is there to say.  Really.  Can you ask specific
>>> questions?
>> Question one is: do you still stand by that earlier assessment?
> Which one?

That there is a good way to do it without explicit support from major modes.

prog-widen would require that support.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Dmitry Gutov]

On 15.11.2019 11:36, Eli Zaretskii wrote:
>> In any case, changing the narrow-to-region so it doesn't change
>> point-min is the right fix.  Adding a call to `jit-lock-fontify-now` is
>> another but worse way to fix it.
> I don't think any of these two ways were proposed as patches, so I
> don't think I understand well enough what are those changes you prefer
> here.

I have already pushed the first of these two ways to master as 
6a2a371c288a13060c44fa1af3ab73db611705a4.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 15 Nov 2019 12:24:00 +0200
> 
> On 15.11.2019 11:31, Eli Zaretskii wrote:
> >>>>>>> The reality could be different,
> >>>>>>> and I'd like us to support those different conditions.  It doesn't
> >>>>>>> sound like it's much harder.
> >>>>>> There are options, sure. But I think all of them will require explicit
> >>>>>> support from major modes.
> >>>>> I don't think so.  All we need to make sure is that the narrowing
> >>>>> doesn't conceal portions of the embedded fragment.
> >>>> I would very much like to read your idea in more detail.
> >>> I'm not sure what else is there to say.  Really.  Can you ask specific
> >>> questions?
> >> Question one is: do you still stand by that earlier assessment?
> > Which one?
> 
> That there is a good way to do it without explicit support from major modes.
> 
> prog-widen would require that support.

I guess it was my misunderstanding of what you meant by "explicit
support".  Sorry.

Re: Several Major Modes.

[Alan Mackenzie]

Hello, Dmitry.

On Fri, Nov 15, 2019 at 00:11:17 +0200, Dmitry Gutov wrote:
> On 14.11.2019 23:24, Alan Mackenzie wrote:
> > Hello, Dmitry.

> > On Thu, Nov 14, 2019 at 00:33:33 +0200, Dmitry Gutov wrote:

> > [ .... ]

> >>>> In mmm-mode context, however, we apply definite boundaries to the
> >>>> region chunks. Here's an example of some Noweb code:
> >>>> https://en.wikipedia.org/wiki/Noweb#Example_of_a_simple_noweb_program

> >>>> The inside of hello.c block would be narrowed to.

> >>> I think I've said this before, but I don't think narrowing is the right
> >>> tool for that task.  I don't think there is a suitable tool in Emacs at
> >>> the moment.

> >> *shrug* We do the best with what we have.

> > Why can we not formulate something better, an enhancement to the Emacs
> > core which would support several major modes properly?  I have made
> > proposals in this area before, but I think they were to grandiose to be
> > implementable.

> > What seems to be needed is a way of partioning a buffer into several
> > sub-buffers (which I have called "islands" in the past), and having a new
> > type of local variable, one valid in exactly one sub-buffer.  More or
> > less.

> There are options. We'd have to decide on a suitable model, calling them 
> islands or whatever, but I think the first approximation is to either 
> make sure narrowing is available for this purpose ....

You also need to make sure narrowing is available for any purpose
required by a major mode.

> .... or, if we absolutely can't make it work, add a new element to
> prog-indentation-context which will be a function that would return the
> bounds of the current chunk.

Or something like that.

> Regarding "new type of local variable", mmm-mode already tracks 
> something like that.

I was envisaging something at the C level, where different regions of a
buffer would have different values of variables, without needing the
continual swapping at the Lisp level.  Maybe such a thing isn't needed.

> >>>> Now, I have remembered that CC Mode calls widen from many places
> >>>> already, so it already is problematic for using in a context like
> >>>> that.

> >>> It does, yes.  Users also use widening and narrowing.

> > I believe these problems won't go away, and there will always be
> > conflicts between CC Mode (as it is) and mmm-mode (as it is).

> I think we should also try to understand whether making CC Mode play 
> nice to doable/feasible, and for what uses. Like, I think it can work 
> (more or less) already when it's the primary major mode (meaning the 
> buffer starts and ends with it), so the embedded chunks are all other 
> modes.

It can't work if any external Lisp corrupts its syntax-table text
properties.  This is what syntax-ppss-flush-cache (on
before-change-functions for many modes) would do if there were a non-nil
syntax-propertize-function at the time.  This may be the biggest problem
to getting CC Mode integrated into MMM Mode.

> Is it feasible to support embedded chunks? To support chunks with
> incomplete pieces of code (which are e.g. included conditionally by the
> surrounding template)?

Well CC Mode already supports preprocessor macros and (for C++) raw
strings, which are syntactically somewhat and very different from the
enclosing code.  So I don't see why not, once the problem with the
syntax-table text properties is solved.

> By answering these questions we can temper our expectations and come up
> with a practical plan.

> Doing nothing is also a valid choice, BTW, since for many uses replacing 
> c-mode with js-mode works pretty okay. I've been recommending it to 
> MMM-mode users.

That's a rather rough workaround.  It would be better if we could solve
the problems.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Several Major Modes.

[Dmitry Gutov]

Hi Alan,

On 15.11.2019 22:10, Alan Mackenzie wrote:

>> There are options. We'd have to decide on a suitable model, calling them
>> islands or whatever, but I think the first approximation is to either
>> make sure narrowing is available for this purpose ....
> 
> You also need to make sure narrowing is available for any purpose
> required by a major mode.

Eh, I think it's "available" already, but I'd have to see specific examples.

The problem which triggered this discussion is that *something* called 
font-lock rules from a narrowed buffer directly. But that's not a 
"purpose required by a major mode".

>> Regarding "new type of local variable", mmm-mode already tracks
>> something like that.
> 
> I was envisaging something at the C level, where different regions of a
> buffer would have different values of variables, without needing the
> continual swapping at the Lisp level.  Maybe such a thing isn't needed.

I'd been told that even a C-based implementation is unlikely to make 
things much faster. Anyway, it would be a perf optimization, and we 
could get to it later.

> It can't work if any external Lisp corrupts its syntax-table text
> properties.  This is what syntax-ppss-flush-cache (on
> before-change-functions for many modes) would do if there were a non-nil
> syntax-propertize-function at the time.  This may be the biggest problem
> to getting CC Mode integrated into MMM Mode.

mmm-mode sets its own syntax-propertize-function that calls major mode 
specific syntax-propertize-function's over their respective 
chunks/subregions. So, in principle, that should work fine. As long as 
nobody calls 'widen' unexpectedly.

>> Is it feasible to support embedded chunks? To support chunks with
>> incomplete pieces of code (which are e.g. included conditionally by the
>> surrounding template)?
> 
> Well CC Mode already supports preprocessor macros and (for C++) raw
> strings, which are syntactically somewhat and very different from the
> enclosing code.

I'm not sure it's the same. Like, would CC Mode cope with a region 
starting with closing brackets, etc. This might not be a frequent 
situation, but at least it shouldn't blow up.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Dmitry Gutov]

On 13.11.2019 23:19, Alan Mackenzie wrote:
>      (ii) This latter function uses vertical-motion to count the lines.
>      (iii) vertical-motion triggers jit-lock fontification.
>      (iv) This calls (eventually) c-font-lock-fontify-region.

By the way, I had a thought.

Like, normally font-lock does widen the buffer before fontifying 
anything. So I was wondering, why doesn't this happen between steps iii 
and iv somewhere?

Turns out, the (unless font-lock-dont-widen (widen)) calls are performed 
inside the default values of font-lock-fontify-region-function and 
font-lock-fontify-buffer-function.

But CC Mode sets font-lock-fontify-region-function to a different value, 
and that specific function doesn't include this piece of code. Or 
rather, didn't. The new commit added an unconditional (widen) call.

I think if you simply wrap it in (unless font-lock-dont-widen ...), it 
will improve compatibility with mmm-mode already (without adverse 
effects, I imagine). After that, you could audit if the other (widen) 
calls spread across CC Mode are still necessary. Maybe they're used in 
indentation code (that's also solvable). Maybe also by some other commands.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Stefan Monnier]

>> IIUC this can't be fixed in font-lock or jit-lock.
> Why not?

Good question.  Looks like I was wrong.
I think the patch below is a better general solution.


        Stefan


diff --git a/lisp/font-lock.el b/lisp/font-lock.el
index 01c19e6e87..9c1c193eda 100644
--- a/lisp/font-lock.el
+++ b/lisp/font-lock.el
@@ -1071,7 +1071,9 @@ font-lock-fontify-region
 If LOUDLY is non-nil, print status messages while fontifying.
 This works by calling `font-lock-fontify-region-function'."
   (font-lock-set-defaults)
-  (funcall font-lock-fontify-region-function beg end loudly))
+  (save-restriction
+    (unless font-lock-dont-widen (widen))
+    (funcall font-lock-fontify-region-function beg end loudly)))
 
 (defun font-lock-unfontify-region (beg end)
   "Unfontify the text between BEG and END.
@@ -1221,8 +1221,6 @@
   (save-buffer-state
     ;; Use the fontification syntax table, if any.
     (with-syntax-table (or font-lock-syntax-table (syntax-table))
-      (save-restriction
-        (unless font-lock-dont-widen (widen))
         ;; Extend the region to fontify so that it starts and ends at
         ;; safe places.
         (let ((funs font-lock-extend-region-functions)
@@ -1251,7 +1249,7 @@
         (unless font-lock-keywords-only
           (font-lock-fontify-syntactically-region beg end loudly))
         (font-lock-fontify-keywords-region beg end loudly)
-        `(jit-lock-bounds ,beg . ,end)))))
+     `(jit-lock-bounds ,beg . ,end))))
 
 ;; The following must be rethought, since keywords can override fontification.
 ;;    ;; Now scan for keywords, but not if we are inside a comment now.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 15 Nov 2019 18:27:21 -0500
> Cc: acm@muc.de, emacs-devel@gnu.org, dgutov@yandex.ru
> 
> >> IIUC this can't be fixed in font-lock or jit-lock.
> > Why not?
> 
> Good question.  Looks like I was wrong.
> I think the patch below is a better general solution.

If it solves the original problem, I like it better as well.

Thanks.

Re: Several Major Modes.

[Alan Mackenzie]

Hello, Dmitry.

On Fri, Nov 15, 2019 at 23:45:44 +0200, Dmitry Gutov wrote:
> On 15.11.2019 22:10, Alan Mackenzie wrote:

> >> There are options. We'd have to decide on a suitable model, calling
> >> them islands or whatever, but I think the first approximation is to
> >> either make sure narrowing is available for this purpose ....

> > You also need to make sure narrowing is available for any purpose
> > required by a major mode.

> Eh, I think it's "available" already, but I'd have to see specific examples.

Oh good!  I have to admit I haven't actually seen MMM Mode running, or
at least I don't remember it.

> The problem which triggered this discussion is that *something* called 
> font-lock rules from a narrowed buffer directly. But that's not a 
> "purpose required by a major mode".

> >> Regarding "new type of local variable", mmm-mode already tracks
> >> something like that.

> > I was envisaging something at the C level, where different regions of a
> > buffer would have different values of variables, without needing the
> > continual swapping at the Lisp level.  Maybe such a thing isn't needed.

> I'd been told that even a C-based implementation is unlikely to make 
> things much faster. Anyway, it would be a perf optimization, and we 
> could get to it later.

OK.

> > It can't work if any external Lisp corrupts its syntax-table text
> > properties.  This is what syntax-ppss-flush-cache (on
> > before-change-functions for many modes) would do if there were a non-nil
> > syntax-propertize-function at the time.  This may be the biggest problem
> > to getting CC Mode integrated into MMM Mode.

> mmm-mode sets its own syntax-propertize-function that calls major mode 
> specific syntax-propertize-function's over their respective 
> chunks/subregions. So, in principle, that should work fine. As long as 
> nobody calls 'widen' unexpectedly.

I've taken a closer look at the functions in syntax.el, and I think
you're right.  The removal of the syntax-table text properties happens
up to END, not to EOB.  It would merely need syntax-propertize-function
to be nil whenever anything involving the CC Mode region (including
fontification) happens.

> >> Is it feasible to support embedded chunks? To support chunks with
> >> incomplete pieces of code (which are e.g. included conditionally by the
> >> surrounding template)?

> > Well CC Mode already supports preprocessor macros and (for C++) raw
> > strings, which are syntactically somewhat and very different from the
> > enclosing code.

> I'm not sure it's the same. Like, would CC Mode cope with a region 
> starting with closing brackets, etc. This might not be a frequent 
> situation, but at least it shouldn't blow up.

Maybe having several sets of syntax-table text properties in a buffer,
one set for each sub-buffer, would help.  I devised and half-implemented
such a facility back in 2017, calling it "indirect text properties".  To
switch to a different set of properties, you would merely have to set
(or bind) a dynamic variable.

With this, I could set whitespace syntax-table props all over the non-CC
Mode regions while CC Mode is "in scope", thus making syntactic stuff
and fontification easy.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Stefan Monnier]

> If it solves the original problem, I like it better as well.

Do you want me to install it, then?


        Stefan

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: acm@muc.de,  emacs-devel@gnu.org,  dgutov@yandex.ru
> Date: Sat, 16 Nov 2019 19:50:13 -0500
> 
> > If it solves the original problem, I like it better as well.
> 
> Do you want me to install it, then?

Unless you see any potential problems with it, sure.

Thanks.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Dmitry Gutov]

On 16.11.2019 1:27, Stefan Monnier wrote:
> Good question.  Looks like I was wrong.
> I think the patch below is a better general solution.

I think it's generally better solution thah what I suggested, but:

- Without reverting the change in the commit mentioned in the subject, 
the situation for CC Mode will be no better.
- If Alan wanted to improve said situation, and if CC Mode still has a 
version distributed separately for previous versions of Emacs, my change 
would be better for it.

The two patches could coexist, though. Despite being largely redundant.

Re: Several Major Modes.

[Dmitry Gutov]

Hi Alan,

On 16.11.2019 15:10, Alan Mackenzie wrote:

>>> You also need to make sure narrowing is available for any purpose
>>> required by a major mode.
> 
>> Eh, I think it's "available" already, but I'd have to see specific examples.
> 
> Oh good!  I have to admit I haven't actually seen MMM Mode running, or
> at least I don't remember it.

I think we won't really know until there's a good change of CC Mode 
working with MMM Mode, so then people could try and exercise various 
features. And we'll see what works and what does not.

> I've taken a closer look at the functions in syntax.el, and I think
> you're right.  The removal of the syntax-table text properties happens
> up to END, not to EOB.  It would merely need syntax-propertize-function
> to be nil whenever anything involving the CC Mode region (including
> fontification) happens.

Yes, it should do that already.

>>>> Is it feasible to support embedded chunks? To support chunks with
>>>> incomplete pieces of code (which are e.g. included conditionally by the
>>>> surrounding template)?
> 
>>> Well CC Mode already supports preprocessor macros and (for C++) raw
>>> strings, which are syntactically somewhat and very different from the
>>> enclosing code.
> 
>> I'm not sure it's the same. Like, would CC Mode cope with a region
>> starting with closing brackets, etc. This might not be a frequent
>> situation, but at least it shouldn't blow up.
> 
> Maybe having several sets of syntax-table text properties in a buffer,
> one set for each sub-buffer, would help.  I devised and half-implemented
> such a facility back in 2017, calling it "indirect text properties".  To
> switch to a different set of properties, you would merely have to set
> (or bind) a dynamic variable.
> 
> With this, I could set whitespace syntax-table props all over the non-CC
> Mode regions while CC Mode is "in scope", thus making syntactic stuff
> and fontification easy.

It's an interesting suggestion, but a difficult problem. Like, if you 
just swap the meanings of syntax-table text properties, it would make 
syntax-ppss caches invalid without telling it. A better solution would 
be to somehow integrate with it.

Anyway, you mentioned a real problem (for which I only have workarounds 
thus far), but for now I was just asking whether an isolated 
"incomplete" chunk would work okay.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Eli Zaretskii]

> Cc: acm@muc.de, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 17 Nov 2019 09:36:53 +0200
> 
> On 16.11.2019 1:27, Stefan Monnier wrote:
> > Good question.  Looks like I was wrong.
> > I think the patch below is a better general solution.
> 
> I think it's generally better solution thah what I suggested, but:
> 
> - Without reverting the change in the commit mentioned in the subject, 
> the situation for CC Mode will be no better.

I thought that commit could now be reverted?

Re: Several Major Modes.

[Alan Mackenzie]

Hello, Dmitry.

On Sun, Nov 17, 2019 at 14:48:35 +0200, Dmitry Gutov wrote:
> Hi Alan,

> On 16.11.2019 15:10, Alan Mackenzie wrote:

> >>> You also need to make sure narrowing is available for any purpose
> >>> required by a major mode.

> >> Eh, I think it's "available" already, but I'd have to see specific
> >> examples.

> > Oh good!  I have to admit I haven't actually seen MMM Mode running,
> > or at least I don't remember it.

> I think we won't really know until there's a good change of CC Mode
> working with MMM Mode, so then people could try and exercise various
> features. And we'll see what works and what does not.

Other than CC Mode's handling of syntax-table text properties, what
precisely is hindering it working in MMM Mode?  (That question's partly
addressed at myself.)

[ .... ]

> >>>> Is it feasible to support embedded chunks? To support chunks with
> >>>> incomplete pieces of code (which are e.g. included conditionally
> >>>> by the surrounding template)?

> >>> Well CC Mode already supports preprocessor macros and (for C++) raw
> >>> strings, which are syntactically somewhat and very different from the
> >>> enclosing code.

> >> I'm not sure it's the same. Like, would CC Mode cope with a region
> >> starting with closing brackets, etc. This might not be a frequent
> >> situation, but at least it shouldn't blow up.

> > Maybe having several sets of syntax-table text properties in a buffer,
> > one set for each sub-buffer, would help.  I devised and half-implemented
> > such a facility back in 2017, calling it "indirect text properties".  To
> > switch to a different set of properties, you would merely have to set
> > (or bind) a dynamic variable.

> > With this, I could set whitespace syntax-table props all over the non-CC
> > Mode regions while CC Mode is "in scope", thus making syntactic stuff
> > and fontification easy.

> It's an interesting suggestion, but a difficult problem. Like, if you 
> just swap the meanings of syntax-table text properties, it would make 
> syntax-ppss caches invalid without telling it. A better solution would 
> be to somehow integrate with it.

Being more precise, with this idea there would be several sets of s-t
properties present at the same time.  Each would have a different symbol,
e.g. gensyms like syntax-table-13, syntax-table-14, .....  The code in
textprop.c would check the symbol 'syntax-table for a particular symbol
property, which if set would cause it to substitute, say,
syntax-table-13, and read/write that property instead.   Fast, and only a
bit inelegant.

Naturally, all the low-level caches in syntax.c would need to be
synchronised at the same time we bind the thing to syntax-table-13.  The
caches belonging to each individual MMM chunk (such as syntax-ppss
caches) would be OK without change, since they would only be used while
the chunk was current, and hence syntax-table-13 currently in use.

Given how much in Emacs depends on syntax, this scheme might save a lot
of work and a lot of workarounds in MMM Mode.  If MMM Mode could be
responsible for putting space syntax-table-13 properties on all chunks
but the current one, it is possible major modes could be used in MMM Mode
with only minimal adaptations to those major modes.

Implementing this scheme would not be difficult.  (As I said, it's half
done already.)  How would you feel about using it in MMM Mode, and do you
see any difficult problems with it?

> Anyway, you mentioned a real problem (for which I only have workarounds 
> thus far), but for now I was just asking whether an isolated 
> "incomplete" chunk would work okay.

:-)  Sorry, yes.  I think working with an incomplete chunk would be
difficult at the moment.  With all the other chunks "spaced out", as
suggested above, it would be easy, I think.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Dmitry Gutov]

On 17.11.2019 17:44, Eli Zaretskii wrote:
>> Cc: acm@muc.de, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Sun, 17 Nov 2019 09:36:53 +0200
>>
>> On 16.11.2019 1:27, Stefan Monnier wrote:
>>> Good question.  Looks like I was wrong.
>>> I think the patch below is a better general solution.
>>
>> I think it's generally better solution thah what I suggested, but:
>>
>> - Without reverting the change in the commit mentioned in the subject,
>> the situation for CC Mode will be no better.
> 
> I thought that commit could now be reverted?

a) I don't see that in the proposed patch.

b) See item 2 from my previous email: that fix is not very compatible 
with "external" CC Mode.

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 17 Nov 2019 19:59:03 +0200
> 
> b) See item 2 from my previous email: that fix is not very compatible 
> with "external" CC Mode.

What is "external CC Mode"?

Re: master 7362554: Widen around c-font-lock-fontify-region. This fixes bug #38049.

[Dmitry Gutov]

On 17.11.2019 20:26, Eli Zaretskii wrote:
>> Cc: monnier@iro.umontreal.ca, acm@muc.de, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Sun, 17 Nov 2019 19:59:03 +0200
>>
>> b) See item 2 from my previous email: that fix is not very compatible
>> with "external" CC Mode.
> 
> What is "external CC Mode"?

The one that I mentioned in the aforementioned "item 2".

http://cc-mode.sourceforge.net/release.php

Re: Several Major Modes.

[Dmitry Gutov]

On 17.11.2019 18:03, Alan Mackenzie wrote:
> Hello, Dmitry.
> 
> On Sun, Nov 17, 2019 at 14:48:35 +0200, Dmitry Gutov wrote:
>> Hi Alan,
> 
>> On 16.11.2019 15:10, Alan Mackenzie wrote:
> 
>>>>> You also need to make sure narrowing is available for any purpose
>>>>> required by a major mode.
> 
>>>> Eh, I think it's "available" already, but I'd have to see specific
>>>> examples.
> 
>>> Oh good!  I have to admit I haven't actually seen MMM Mode running,
>>> or at least I don't remember it.
> 
>> I think we won't really know until there's a good change of CC Mode
>> working with MMM Mode, so then people could try and exercise various
>> features. And we'll see what works and what does not.
> 
> Other than CC Mode's handling of syntax-table text properties, what
> precisely is hindering it working in MMM Mode?  (That question's partly
> addressed at myself.)
> 
> [ .... ]

Here's the email you missed. For some reason I can't find it in the 
emacs-devel archive, but it was also addressed to you and Stefan, and 
Stefan received it:

(quote begins)

By the way, I had a thought.

Like, normally font-lock does widen the buffer before fontifying 
anything. So I was wondering, why doesn't this happen between steps iii 
and iv somewhere?

Turns out, the (unless font-lock-dont-widen (widen)) calls are performed 
inside the default values of font-lock-fontify-region-function and 
font-lock-fontify-buffer-function.

But CC Mode sets font-lock-fontify-region-function to a different value, 
and that specific function doesn't include this piece of code. Or 
rather, didn't. The new commit added an unconditional (widen) call.

I think if you simply wrap it in (unless font-lock-dont-widen ...), it 
will improve compatibility with mmm-mode already (without adverse 
effects, I imagine). After that, you could audit if the other (widen) 
calls spread across CC Mode are still necessary. Maybe they're used in 
indentation code (that's also solvable). Maybe also by some other commands.

(quote ends)

This would be the first step. It really comes down to my original request:

diff --git a/lisp/progmodes/cc-mode.el b/lisp/progmodes/cc-mode.el
index 73160fc7a4..9b27ee79e9 100644
--- a/lisp/progmodes/cc-mode.el
+++ b/lisp/progmodes/cc-mode.el
@@ -2233,7 +2233,7 @@ c-font-lock-fontify-region
    ;; Type a space in the first blank line, and the fontification of 
the next
    ;; line was fouled up by context fontification.
    (save-restriction
-    (widen)
+    (unless font-lock-dont-widen (widen))
      (let (new-beg new-end new-region case-fold-search)
        (if (and c-in-after-change-fontification
  	       (< beg c-new-END) (> end c-new-BEG))