What to do when font-lock-flush + font-lock-ensure do less than font-lock-fontify-buffer?

What to do when font-lock-flush + font-lock-ensure do less than font-lock-fontify-buffer?

[Clément Pit--Claudel]

[-- Attachment #1: Type: text/plain, Size: 644 bytes --]

Hi emacs-devel,

IIUC the proper way in Emacs 25 to signal changes that require a refontification to font-lock is to call font-lock-flush. This doesn't always do as much as font-lock-fontify-buffer does, though.

In particular, when the font-lock-fontified variable is nil (which does happen, though I don't know why; I don't know what this variable is for), then calling font-lock-fontify buffer does refontify the whole buffer, but font-lock-flush and font-lock-ensure don't seem to do anything.

What's the proper way to mark a buffer for refontification (and refontify the visible portion) immediately in 25?

Thanks,
Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: What to do when font-lock-flush + font-lock-ensure do less than font-lock-fontify-buffer?

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 383 bytes --]

>
> What's the proper way to mark a buffer for refontification (and refontify
> the visible portion) immediately in 25?
>

I've been using `(font-lock-region (point-min) (point-max))' to force a
refontification of the entire buffer, for my htmlize-like packages
(`faceup' and `e2ani'). I'm not sure if it's the "right way" to do it, but
it works in practice.

    -- Anders Lindgren

[-- Attachment #2: Type: text/html, Size: 681 bytes --]

Re: What to do when font-lock-flush + font-lock-ensure do less than font-lock-fontify-buffer?

[Clément Pit--Claudel]

[-- Attachment #1: Type: text/plain, Size: 596 bytes --]

On 02/22/2016 03:56 AM, Anders Lindgren wrote:
>> What's the proper way to mark a buffer for refontification (and
>> refontify the visible portion) immediately in 25?
>>
> I've been using `(font-lock-region (point-min) (point-max))' to force
> a refontification of the entire buffer, for my htmlize-like packages
> (`faceup' and `e2ani'). I'm not sure if it's the "right way" to do
> it, but it works in practice.

Thanks. Won't that refontify the whole buffer though, even the non-visible (and not yet fontified) areas, though? That would be a performance issue for me.

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: What to do when font-lock-flush + font-lock-ensure do less than font-lock-fontify-buffer?

[Anders Lindgren]

[-- Attachment #1: Type: text/plain, Size: 503 bytes --]

>
> > I've been using `(font-lock-region (point-min) (point-max))' to force
> > a refontification of the entire buffer, for my htmlize-like packages
> > (`faceup' and `e2ani'). I'm not sure if it's the "right way" to do
> > it, but it works in practice.
>
> Thanks. Won't that refontify the whole buffer though, even the non-visible
> (and not yet fontified) areas, though? That would be a performance issue
> for me.
>

Yes, it does, so I guess this is not the correct solution for you.

    -- Anders

[-- Attachment #2: Type: text/html, Size: 849 bytes --]

Re: What to do when font-lock-flush + font-lock-ensure do less than font-lock-fontify-buffer?

[Eli Zaretskii]

> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Sun, 21 Feb 2016 23:55:04 -0500
> 
> IIUC the proper way in Emacs 25 to signal changes that require a refontification to font-lock is to call font-lock-flush. This doesn't always do as much as font-lock-fontify-buffer does, though.
> 
> In particular, when the font-lock-fontified variable is nil (which does happen, though I don't know why; I don't know what this variable is for), then calling font-lock-fontify buffer does refontify the whole buffer, but font-lock-flush and font-lock-ensure don't seem to do anything.
> 
> What's the proper way to mark a buffer for refontification (and refontify the visible portion) immediately in 25?

I think what you describe _is_ the proper way.  Can you show a test
case which demonstrates how calling font-lock-flush followed by
font-lock-ensure fails to do the job?

Thanks.

Re: What to do when font-lock-flush + font-lock-ensure do less than font-lock-fontify-buffer?

[Clément Pit--Claudel]

[-- Attachment #1: Type: text/plain, Size: 1215 bytes --]

On 02/22/2016 11:13 AM, Eli Zaretskii wrote:
>> From: Clément Pit--Claudel <clement.pit@gmail.com>
>> Date: Sun, 21 Feb 2016 23:55:04 -0500
>>
>> IIUC the proper way in Emacs 25 to signal changes that require a refontification to font-lock is to call font-lock-flush. This doesn't always do as much as font-lock-fontify-buffer does, though.
>>
>> In particular, when the font-lock-fontified variable is nil (which does happen, though I don't know why; I don't know what this variable is for), then calling font-lock-fontify buffer does refontify the whole buffer, but font-lock-flush and font-lock-ensure don't seem to do anything.
>>
>> What's the proper way to mark a buffer for refontification (and refontify the visible portion) immediately in 25?
> 
> I think what you describe _is_ the proper way.  Can you show a test
> case which demonstrates how calling font-lock-flush followed by
> font-lock-ensure fails to do the job?

In any case where font-lock-fontified is nil font-lock-fontify-buffer will properly refontify the whole buffer, but font-lock-flush and font-lock ensure will not do anything. I don't know what causes that variable to be nil in the first place, though.

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: What to do when font-lock-flush + font-lock-ensure do less than font-lock-fontify-buffer?

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 1116 bytes --]

On Mon, Feb 22, 2016 at 11:29 AM, Clément Pit--Claudel <
clement.pit@gmail.com> wrote:

> In any case where font-lock-fontified is nil font-lock-fontify-buffer will
> properly refontify the whole buffer, but font-lock-flush and font-lock
> ensure will not do anything. I don't know what causes that variable to be
> nil in the first place, though.


I am not sure if this is related. But in order for hi-lock-file-patterns to
work in a text-mode buffer, I need to do the below because
font-lock-fontified is nil for text-mode (as technically there are no
comments or keywords to highlight in that major mode).

    ;; Enable `hi-lock-mode' in `text-mode' too
    ;; The hi-lock fontification will not be visible (the
`font-lock-keywords'
    ;; variable will not be updated unless `font-lock-fontified' is already
`t'.
    ;; This was derived by studying the definition of
`hi-lock-font-lock-hook'
    ;; function.
    (defun modi/hi-lock-enable-in-text-mode ()
      (setq-local font-lock-fontified t))
    (add-hook 'text-mode-hook #'modi/hi-lock-enable-in-text-mode)

--
Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 1939 bytes --]

Re: What to do when font-lock-flush + font-lock-ensure do less than font-lock-fontify-buffer?

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Mon, 22 Feb 2016 11:29:10 -0500
> 
> > I think what you describe _is_ the proper way.  Can you show a test
> > case which demonstrates how calling font-lock-flush followed by
> > font-lock-ensure fails to do the job?
> 
> In any case where font-lock-fontified is nil
> font-lock-fontify-buffer will properly refontify the whole buffer,
> but font-lock-flush and font-lock ensure will not do anything.

That's not what I see in the code, that's why I asked for a test
case.  I'm probably missing something.

Re: What to do when font-lock-flush + font-lock-ensure do less than font-lock-fontify-buffer?

[Clément Pit--Claudel]

[-- Attachment #1: Type: text/plain, Size: 1297 bytes --]

On 02/22/2016 11:40 AM, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org
>> From: Clément Pit--Claudel <clement.pit@gmail.com>
>> Date: Mon, 22 Feb 2016 11:29:10 -0500
>>
>>> I think what you describe _is_ the proper way.  Can you show a test
>>> case which demonstrates how calling font-lock-flush followed by
>>> font-lock-ensure fails to do the job?
>>
>> In any case where font-lock-fontified is nil
>> font-lock-fontify-buffer will properly refontify the whole buffer,
>> but font-lock-flush and font-lock ensure will not do anything.
> 
> That's not what I see in the code, that's why I asked for a test
> case.  I'm probably missing something.

Here's a test case:

(with-current-buffer (get-buffer-create "temp")
  (erase-buffer)
  (setq-default prettify-symbols-unprettify-at-point 'right-edge)
  (emacs-lisp-mode)
  (prettify-symbols-mode)
  (insert "lambda\n\nlambda")
  (pop-to-buffer (current-buffer)))

Moving around in this buffer unprettifies and reprettifies ‘lambda’s into ‘λ’s as the point moves in and out of them. Setting ‘font-lock-fontified’ to nil breaks that mechanism (the ‘λ’s are unprettified but no reprettified), and no series of ‘font-lock-flush’ and ‘font-lock-ensure’ will reprettify them.

Does this help?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: What to do when font-lock-flush + font-lock-ensure do less than font-lock-fontify-buffer?

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Mon, 22 Feb 2016 12:28:13 -0500
> 
> (with-current-buffer (get-buffer-create "temp")
>   (erase-buffer)
>   (setq-default prettify-symbols-unprettify-at-point 'right-edge)
>   (emacs-lisp-mode)
>   (prettify-symbols-mode)
>   (insert "lambda\n\nlambda")
>   (pop-to-buffer (current-buffer)))
> 
> Moving around in this buffer unprettifies and reprettifies ‘lambda’s into ‘λ’s as the point moves in and out of them. Setting ‘font-lock-fontified’ to nil breaks that mechanism (the ‘λ’s are unprettified but no reprettified), and no series of ‘font-lock-flush’ and ‘font-lock-ensure’ will reprettify them.
> 
> Does this help?

Yes, thanks.  I will look into this soon.

Re: What to do when font-lock-flush + font-lock-ensure do less than font-lock-fontify-buffer?

[Clément Pit--Claudel]

[-- Attachment #1: Type: text/plain, Size: 920 bytes --]

On 02/23/2016 11:57 AM, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org
>> From: Clément Pit--Claudel <clement.pit@gmail.com>
>> Date: Mon, 22 Feb 2016 12:28:13 -0500
>>
>> (with-current-buffer (get-buffer-create "temp")
>>   (erase-buffer)
>>   (setq-default prettify-symbols-unprettify-at-point 'right-edge)
>>   (emacs-lisp-mode)
>>   (prettify-symbols-mode)
>>   (insert "lambda\n\nlambda")
>>   (pop-to-buffer (current-buffer)))
>>
>> Moving around in this buffer unprettifies and reprettifies ‘lambda’s into ‘λ’s as the point moves in and out of them. Setting ‘font-lock-fontified’ to nil breaks that mechanism (the ‘λ’s are unprettified but no reprettified), and no series of ‘font-lock-flush’ and ‘font-lock-ensure’ will reprettify them.
>>
>> Does this help?
> 
> Yes, thanks.  I will look into this soon.

Great; thank you very much. Let me know if I can help!


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: What to do when font-lock-flush + font-lock-ensure do less than font-lock-fontify-buffer?

[Stefan Monnier]

> In particular, when the font-lock-fontified variable is nil (which does
> happen, though I don't know why; I don't know what this variable is for),

AFAIK this only happens when font-lock-mode is OFF, or when jit-lock is
not used and the buffer is larger than font-lock-maximum-size.

> then calling font-lock-fontify buffer does refontify the whole buffer, but
> font-lock-flush and font-lock-ensure don't seem to do anything.

It's normal for font-lock-flush to do nothing.  As for font-lock-ensure
it's a bug which I believe I fixed a few weeks ago.


        Stefan

Re: What to do when font-lock-flush + font-lock-ensure do less than font-lock-fontify-buffer?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 23 Feb 2016 22:52:38 -0500
> 
> > In particular, when the font-lock-fontified variable is nil (which does
> > happen, though I don't know why; I don't know what this variable is for),
> 
> AFAIK this only happens when font-lock-mode is OFF, or when jit-lock is
> not used and the buffer is larger than font-lock-maximum-size.
> 
> > then calling font-lock-fontify buffer does refontify the whole buffer, but
> > font-lock-flush and font-lock-ensure don't seem to do anything.
> 
> It's normal for font-lock-flush to do nothing.  As for font-lock-ensure
> it's a bug which I believe I fixed a few weeks ago.

Actually, in this particular case I think this is a bug in
prettify-symbols-mode: it should define its own
font-lock-ensure-function.  The default value doesn't cover the kind
of stuff that this mode uses font-lock for.

Re: What to do when font-lock-flush + font-lock-ensure do less than font-lock-fontify-buffer?

[Stefan Monnier]

> Actually, in this particular case I think this is a bug in
> prettify-symbols-mode: it should define its own
> font-lock-ensure-function.  The default value doesn't cover the kind
> of stuff that this mode uses font-lock for.

Hmm... I don't see any reason why that should be the case.
IOW I think that if a change is needed it should be font-lock.el because
prettify-symbols-mode seems to be using font-lock in a "normal" way.


        Stefan

Re: What to do when font-lock-flush + font-lock-ensure do less than font-lock-fontify-buffer?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: emacs-devel@gnu.org
> Date: Wed, 24 Feb 2016 13:55:42 -0500
> 
> > Actually, in this particular case I think this is a bug in
> > prettify-symbols-mode: it should define its own
> > font-lock-ensure-function.  The default value doesn't cover the kind
> > of stuff that this mode uses font-lock for.
> 
> Hmm... I don't see any reason why that should be the case.
> IOW I think that if a change is needed it should be font-lock.el because
> prettify-symbols-mode seems to be using font-lock in a "normal" way.

I'm not sure about this (this machinery and its implications were
never sufficiently documented, and what I think I know is the result
of a desperate attempt to document it).  However, please explain why
font-lock-ensure-function calls font-lock-DEFAULT-fontify-buffer.  If
it's supposed to be good for any mode, why does it call the default
fontification?

As a matter of fact, if I replace the font-lock-default-fontify-buffer
call with a call to font-lock-fontify-buffer, the problem described by
the OP goes away.  That is why I said what I said.

Re: What to do when font-lock-flush + font-lock-ensure do less than font-lock-fontify-buffer?

[Stefan Monnier]

> However, please explain why font-lock-ensure-function calls
> font-lock-DEFAULT-fontify-buffer.

Good question.

> As a matter of fact, if I replace the font-lock-default-fontify-buffer
> call with a call to font-lock-fontify-buffer, the problem described by
> the OP goes away.  That is why I said what I said.

I haven't seen the beginning of the thread, but if that fixes the
problem, then I think it's a good change.

To go back to the previous question.  I guess I used
font-lock-default-fontify-buffer because the whole point is to get rid
of all the misuses of font-lock-fontify-buffer, but I think in this case
it's indeed the right thing to do.


        Stefan

Re: What to do when font-lock-flush + font-lock-ensure do less than font-lock-fontify-buffer?

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 65 bytes --]

Hi Eli,

Should I open a bug report about this?

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: What to do when font-lock-flush + font-lock-ensure do less than font-lock-fontify-buffer?

[Eli Zaretskii]

> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Sat, 9 Apr 2016 14:59:39 -0400
> 
> Should I open a bug report about this?

If you have a patch, it would be more efficient to just post it.

Thanks.

Re: What to do when font-lock-flush + font-lock-ensure do less than font-lock-fontify-buffer?

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 556 bytes --]

On 04/09/2016 03:37 PM, Eli Zaretskii wrote:
>> From: Clément Pit--Claudel <clement.pit@gmail.com>
>> Date: Sat, 9 Apr 2016 14:59:39 -0400
>>
>> Should I open a bug report about this?
> 
> If you have a patch, it would be more efficient to just post it.

I don't have one, unfortunately; I think you understand the issue better than I do :) 
You mentioned turning a call to font-lock-default-fontify-buffer into font-lock-fontify-buffer as a potential solution earlier in this thread. Do you want me to make a patch that does this?

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: What to do when font-lock-flush + font-lock-ensure do less than font-lock-fontify-buffer?

[Stefan Monnier]

> Should I open a bug report about this?

Yes.  And put me in X-Debbugs-Cc.


        Stefan

RE: What to do when font-lock-flush + font-lock-ensure do less than font-lock-fontify-buffer?

[Drew Adams]

> I'm not sure about this (this machinery and its implications were
> never sufficiently documented, and what I think I know is the result
> of a desperate attempt to document it).

File a doc bug. ;-)