bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Ihor Radchenko]

Hi,

font-lock-mode contains the follow clause:

  ;; Don't turn on Font Lock mode if we don't have a display (we're running a
  ;; batch job) or if the buffer is invisible (the name starts with a space).
  (when (or noninteractive (eq (aref (buffer-name) 0) ?\s))
    (setq font-lock-mode nil))

which interferes with Org mode's fontification code in some edge cases.

Org mode fontifies code blocks using "hidden" temporary buffers with
their name starting from " ": " *org-src-fontification:major-mode*".

The buffers are kept around to avoid extra major-mode loading overheads
and the fontification is not using (font-lock-ensure).

(font-lock-ensure) usually fontifies the buffer correctly even though
font-lock-mode is disabled. However, not always.

We just got some edge case
https://teddit.net/r/orgmode/comments/w2b0tw/syntax_highlighting_in_orgsource_blocks/igqdx18/
with web-mode, which directly sets 'font-lock-face text properties.

Without font-lock-mode being enabled, 'font-lock-face is not
automatically remapped to 'face text property and hence some part of the
fontification may not be reflected by simply looking into 'face
property.

While Org can work around this particular case by looking into
'font-lock-face, I am not sure if disabled font-lock may not trigger
similar issues in other cases.

It would be nice if we could somehow enable font-lock inside " *hidden*"
buffers. Would it be possible?

Best,
Ihor

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Date: Tue, 19 Jul 2022 12:15:33 +0800
> 
> font-lock-mode contains the follow clause:
> 
>   ;; Don't turn on Font Lock mode if we don't have a display (we're running a
>   ;; batch job) or if the buffer is invisible (the name starts with a space).
>   (when (or noninteractive (eq (aref (buffer-name) 0) ?\s))
>     (setq font-lock-mode nil))
> 
> which interferes with Org mode's fontification code in some edge cases.
> 
> Org mode fontifies code blocks using "hidden" temporary buffers with
> their name starting from " ": " *org-src-fontification:major-mode*".
> 
> The buffers are kept around to avoid extra major-mode loading overheads
> and the fontification is not using (font-lock-ensure).
> 
> (font-lock-ensure) usually fontifies the buffer correctly even though
> font-lock-mode is disabled. However, not always.
> 
> We just got some edge case
> https://teddit.net/r/orgmode/comments/w2b0tw/syntax_highlighting_in_orgsource_blocks/igqdx18/
> with web-mode, which directly sets 'font-lock-face text properties.
> 
> Without font-lock-mode being enabled, 'font-lock-face is not
> automatically remapped to 'face text property and hence some part of the
> fontification may not be reflected by simply looking into 'face
> property.
> 
> While Org can work around this particular case by looking into
> 'font-lock-face, I am not sure if disabled font-lock may not trigger
> similar issues in other cases.
> 
> It would be nice if we could somehow enable font-lock inside " *hidden*"
> buffers. Would it be possible?

Depends on the "somehow" part.  I generally prefer that obscure corner
cases be solved in the software which creates those obscure cases.  In
this case, it seems like web-mode is that place?  I'm not sure why
you say Org should solve this, but perhaps I didn't understand the
description well enough.  But if this indeed needs to be solved in
Org, how about not using buffers whose names begin with a space?

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Org mode fontifies code blocks using "hidden" temporary buffers with
> their name starting from " ": " *org-src-fontification:major-mode*".
>
> The buffers are kept around to avoid extra major-mode loading overheads
> and the fontification is not using (font-lock-ensure).

IIUC they're never displayed, right?  So, `font-lock-ensure` does sound
like "the right tool".  Could you give a short explanation for why you
don't use `font-lock-ensure`?

> (font-lock-ensure) usually fontifies the buffer correctly even though
> font-lock-mode is disabled. However, not always.

Sounds like a bug we should fix.

> It would be nice if we could somehow enable font-lock inside " *hidden*"
> buffers.  Would it be possible?

FWIW, I do think it would be good to remove the rule that font-lock is
never enabled in temp buffers.


        Stefan

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Eli Zaretskii]

> Cc: 56637@debbugs.gnu.org
> Date: Tue, 19 Jul 2022 12:13:29 -0400
> From:  Stefan Monnier via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> FWIW, I do think it would be good to remove the rule that font-lock is
> never enabled in temp buffers.

I don't.

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> FWIW, I do think it would be good to remove the rule that font-lock is
>> never enabled in temp buffers.
> I don't.

I think we agree :-)

More seriously, I didn't mean to remove it willy-nilly, but that it's
a desirable goal and the question is how to get there.

We could start by adding a new variable
`font-lock-allow-in-temporary-buffer`, which packages could set
buffer-locally in those rare temp buffers where they do want to enable
font-lock, as in the untested patch below.

Of course, the problem in `font-lock-ensure` is orthogonal to this.


        Stefan


diff --git a/lisp/font-core.el b/lisp/font-core.el
index f92d1e38306..f74d1d854a0 100644
--- a/lisp/font-core.el
+++ b/lisp/font-core.el
@@ -78,6 +78,13 @@ font-lock-function
 ;; The mode for which font-lock was initialized, or nil if none.
 (defvar font-lock-major-mode)
 
+(defvar-local font-lock-allow-in-temporary-buffer nil
+  "If non-nil, override the temp-buffer limitation.
+Normally `font-lock-mode' cannot be enabled in buffers never intended
+to be displayed (i.e. temp buffers and batch mode).
+Set this variable to non-nil in those rare buffers where `font-lock-mode'
+is needed even though they'll never be displayed.")
+
 (define-minor-mode font-lock-mode
   "Toggle syntax highlighting in this buffer (Font Lock mode).
 
@@ -131,13 +138,14 @@ font-lock-mode
   :after-hook (font-lock-initial-fontify)
   ;; Don't turn on Font Lock mode if we don't have a display (we're running a
   ;; batch job) or if the buffer is invisible (the name starts with a space).
-  (when (or noninteractive (eq (aref (buffer-name) 0) ?\s))
+  (when (and (or noninteractive (eq (aref (buffer-name) 0) ?\s))
+             (not font-lock-allow-in-temporary-buffers))
     (setq font-lock-mode nil))
   (funcall font-lock-function font-lock-mode)
   ;; Arrange to unfontify this buffer if we change major mode later.
   (if font-lock-mode
-      (add-hook 'change-major-mode-hook 'font-lock-change-mode nil t)
-    (remove-hook 'change-major-mode-hook 'font-lock-change-mode t)))
+      (add-hook 'change-major-mode-hook #'font-lock-change-mode nil t)
+    (remove-hook 'change-major-mode-hook #'font-lock-change-mode t)))
 
 ;; Get rid of fontification for the old major mode.
 ;; We do this when changing major modes.

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> More seriously, I didn't mean to remove it willy-nilly, but that it's
> a desirable goal and the question is how to get there.

Of course, for that we need to know what effect this code has in
practice, so I tried the patch below a the first few messages I got
were:

    Enabling font-lock for mode minibuffer-inactive-mode in hidden buffer #<buffer  *Minibuf-0*>
    Enabling font-lock for mode fundamental-mode in hidden buffer #<buffer  *load*>
    Enabling font-lock for mode fundamental-mode in hidden buffer #<buffer  *Echo Area 0*>
    Enabling font-lock for mode fundamental-mode in hidden buffer #<buffer  *Echo Area 1*>
    Enabling font-lock for mode fundamental-mode in hidden buffer #<buffer  *load*-628722>
    Enabling font-lock for mode checkdoc-output-mode in hidden buffer #<buffer  *checkdoc-temp*>
    Enabling font-lock for mode minibuffer-mode in hidden buffer #<buffer  *Minibuf-1*>
    Enabling font-lock for mode minibuffer-inactive-mode in hidden buffer #<buffer  *Minibuf-0*>
    Enabling font-lock for mode minibuffer-inactive-mode in hidden buffer #<buffer  *Minibuf-1*>

Maybe it would be mostly "harmless" to enable font-lock in those buffers/modes,
but even if it is, it would clearly be wasteful.

Side note: the *checkdoc-temp* seems like an odd case.  Its major mode is
clearly designed for a "normal/visible" buffer (and IIUC it is indeed
used in buffers that are displayed), so I'm not sure how *checkdoc-temp*
ends up using that mode.


        Stefan


diff --git a/lisp/font-core.el b/lisp/font-core.el
index f92d1e38306..5678ab22da5 100644
--- a/lisp/font-core.el
+++ b/lisp/font-core.el
@@ -132,12 +132,15 @@ font-lock-mode
   ;; Don't turn on Font Lock mode if we don't have a display (we're running a
   ;; batch job) or if the buffer is invisible (the name starts with a space).
   (when (or noninteractive (eq (aref (buffer-name) 0) ?\s))
-    (setq font-lock-mode nil))
+    ;; (setq font-lock-mode nil)
+    (when font-lock-mode
+      (message "Enabling font-lock for mode %S in hidden buffer %S"
+               major-mode (current-buffer))))
   (funcall font-lock-function font-lock-mode)
   ;; Arrange to unfontify this buffer if we change major mode later.
   (if font-lock-mode
-      (add-hook 'change-major-mode-hook 'font-lock-change-mode nil t)
-    (remove-hook 'change-major-mode-hook 'font-lock-change-mode t)))
+      (add-hook 'change-major-mode-hook #'font-lock-change-mode nil t)
+    (remove-hook 'change-major-mode-hook #'font-lock-change-mode t)))
 
 ;; Get rid of fontification for the old major mode.
 ;; We do this when changing major modes.
@@ -270,7 +273,7 @@ global-font-lock-mode
   font-lock-mode turn-on-font-lock-if-desired
   ;; What was this :extra-args thingy for?  --Stef
   ;; :extra-args (dummy)
-  :initialize 'custom-initialize-delay
+  :initialize #'custom-initialize-delay
   :init-value (not (or noninteractive emacs-basic-display))
   :group 'font-lock
   :version "22.1")

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: yantar92@gmail.com,  56637@debbugs.gnu.org
> Date: Tue, 19 Jul 2022 13:01:38 -0400
> 
> >> FWIW, I do think it would be good to remove the rule that font-lock is
> >> never enabled in temp buffers.
> > I don't.
> 
> I think we agree :-)
> 
> More seriously, I didn't mean to remove it willy-nilly, but that it's
> a desirable goal and the question is how to get there.

Why is that a desirable goal?  It slows down stuff that is never
displayed.  And if that buffer has very long lines, it slows Emacs
down considerably.  I see no reason whatsoever to pay this price, let
alone the garbage font-lock generates.

From my POV, this has "non-starter" written all over it.

> We could start by adding a new variable
> `font-lock-allow-in-temporary-buffer`, which packages could set
> buffer-locally in those rare temp buffers where they do want to enable
> font-lock, as in the untested patch below.

This is a slippery slope.  Next some other corner case will want to
run buffer-modification hooks in a temporary buffer, so we invent
another variable, and so on and so forth.

I'd rather introduce a macro like with-temp-buffer, but one that can
accept a buffer name, and ask such applications to please name their
temporary buffers something that doesn't begin with a space.

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> >> FWIW, I do think it would be good to remove the rule that font-lock is
>> >> never enabled in temp buffers.
>> > I don't.
>> I think we agree :-)
>> More seriously, I didn't mean to remove it willy-nilly, but that it's
>> a desirable goal and the question is how to get there.
> Why is that a desirable goal?

Because testing `noninteractive` and the presence of a leading space are
just heuristics and they aren't quite right.  IOW it's a handy hack but
it's just a hack.

> It slows down stuff that is never displayed.

That's why I say that we shouldn't do it "willy-nilly".

> And if that buffer has very long lines, it slows Emacs down considerably.

I think currently it should basically never be the case, because the
actual fontification (the part which pays attention to lines) only
happens via jit-lock, which should never trigger if those buffers
aren't displayed.

Also, most of those buffers have no font-lock rules installed so
(font-lock-specified-p mode) should return nil and
`font-lock-default-function` won't call `font-lock-mode-internal`.

But yes, if we do it naively, we'll surely bump into corner cases where
there happens to be font-lock rules setup and where jit-lock is disabled
for some reason, or something else ends up causing some of the font-lock
machinery to be run.

Or maybe just running font-lock's setup code itself ends up too costly
for some reason.

>> We could start by adding a new variable
>> `font-lock-allow-in-temporary-buffer`, which packages could set
>> buffer-locally in those rare temp buffers where they do want to enable
>> font-lock, as in the untested patch below.
> This is a slippery slope.  Next some other corner case will want to
> run buffer-modification hooks in a temporary buffer, so we invent
> another variable, and so on and so forth.

Hmm... we do run buffer-modification hooks in temp buffers.

> I'd rather introduce a macro like with-temp-buffer, but one that can
> accept a buffer name, and ask such applications to please name their
> temporary buffers something that doesn't begin with a space.

The purpose of the leading space, AFAIK is to avoid displaying the
buffer in `read-buffer`.  Forcing those buffers to appear in
`read-buffer` just because we need to run font-lock in there
sounds wrong.

If you don't like `font-lock-allow-in-temporary-buffer`, we could have
a new function `font-lock-enable-unconditionally` which does the same as
`font-lock-mode` but skips the (or noninteractive (eq (aref
(buffer-name) 0) ?\s)) test.


        Stefan

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: yantar92@gmail.com,  56637@debbugs.gnu.org
> Date: Tue, 19 Jul 2022 15:21:22 -0400
> 
> >> >> FWIW, I do think it would be good to remove the rule that font-lock is
> >> >> never enabled in temp buffers.
> >> > I don't.
> >> I think we agree :-)
> >> More seriously, I didn't mean to remove it willy-nilly, but that it's
> >> a desirable goal and the question is how to get there.
> > Why is that a desirable goal?
> 
> Because testing `noninteractive` and the presence of a leading space are
> just heuristics and they aren't quite right.  IOW it's a handy hack but
> it's just a hack.

I agree with its being a heuristic, but disagree that they aren't
quite right, and definitely not that they are a hack.  Its our way to
have cheap temporary buffers, and it works quite well.

> > It slows down stuff that is never displayed.
> 
> That's why I say that we shouldn't do it "willy-nilly".
> 
> > And if that buffer has very long lines, it slows Emacs down considerably.
> 
> I think currently it should basically never be the case, because the
> actual fontification (the part which pays attention to lines) only
> happens via jit-lock, which should never trigger if those buffers
> aren't displayed.

jit-lock _will_ trigger if you invoke some APIs that call display
routines.  And that's just one way font-lock can slow down there.

> >> We could start by adding a new variable
> >> `font-lock-allow-in-temporary-buffer`, which packages could set
> >> buffer-locally in those rare temp buffers where they do want to enable
> >> font-lock, as in the untested patch below.
> > This is a slippery slope.  Next some other corner case will want to
> > run buffer-modification hooks in a temporary buffer, so we invent
> > another variable, and so on and so forth.
> 
> Hmm... we do run buffer-modification hooks in temp buffers.

Sorry, I meant kill-buffer-hook, kill-buffer-query-functions, and
buffer-list-update-hook.

> > I'd rather introduce a macro like with-temp-buffer, but one that can
> > accept a buffer name, and ask such applications to please name their
> > temporary buffers something that doesn't begin with a space.
> 
> The purpose of the leading space, AFAIK is to avoid displaying the
> buffer in `read-buffer`.  Forcing those buffers to appear in
> `read-buffer` just because we need to run font-lock in there
> sounds wrong.

That's fixable, and even if we do that in core (something that I'm not
sure is needed), it is still better than having variables to
selectively enable what we generally want to disable in temporary
buffers.

And then, of course, people are free to write their own macros that
are like with-temp-buffer, but different in some way.  Why do we need
to cater for every corner case in core?

> If you don't like `font-lock-allow-in-temporary-buffer`, we could have
> a new function `font-lock-enable-unconditionally` which does the same as
> `font-lock-mode` but skips the (or noninteractive (eq (aref
> (buffer-name) 0) ?\s)) test.

Ouch!

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> >> We could start by adding a new variable
>> >> `font-lock-allow-in-temporary-buffer`, which packages could set
>> >> buffer-locally in those rare temp buffers where they do want to enable
>> >> font-lock, as in the untested patch below.
>> > This is a slippery slope.  Next some other corner case will want to
>> > run buffer-modification hooks in a temporary buffer, so we invent
>> > another variable, and so on and so forth.
>> Hmm... we do run buffer-modification hooks in temp buffers.
> Sorry, I meant kill-buffer-hook, kill-buffer-query-functions, and
> buffer-list-update-hook.

IIUC these used to be run also in temp buffers, so I don't have yet much
intuition for how likely it is that we'll need to avoid those cases.
But, AFAIK these *are* run in batch mode, so it's a slightly
different situation.

>> > I'd rather introduce a macro like with-temp-buffer, but one that can
>> > accept a buffer name, and ask such applications to please name their
>> > temporary buffers something that doesn't begin with a space.
>> The purpose of the leading space, AFAIK is to avoid displaying the
>> buffer in `read-buffer`.  Forcing those buffers to appear in
>> `read-buffer` just because we need to run font-lock in there
>> sounds wrong.
> That's fixable, and even if we do that in core (something that I'm not
> sure is needed), it is still better than having variables to
> selectively enable what we generally want to disable in temporary
> buffers.

Basically, there is a need to have buffers which have font-lock enabled yet
should be hidden from `read-buffer`.  We could try to fix that on the
side of `internal-complete-buffer`, admittedly.

But there is also a need for buffers with font-lock in batch mode
(typically for testing purposes), so changing `internal-complete-buffer`
is not sufficient.

> And then, of course, people are free to write their own macros that
> are like with-temp-buffer, but different in some way.  Why do we need
> to cater for every corner case in core?

Whether you use `with-temp-buffer` or your own macro, you're still stuck
with the choice between "either can't use font-lock or it shows up in `C-x b`".
Of course, people can use hacks like

   (defun really-turn-on-font-lock ()
     (unwind-protect
         (let ((noninteractive nil))
           (rename-buffer (concat "\0" (buffer-name)))
           (font-lock-mode 1))
       (when (eq ?\0 (aref (buffer-name) 0))
         (rename-buffer (substring (buffer-name) 1)))))

But I can't see why we shouldn't accommodate those needs more directly.
It's not like there a deep technical reason why font-lock should not be
enabled under any circumstance in those cases.

>> If you don't like `font-lock-allow-in-temporary-buffer`, we could have
>> a new function `font-lock-enable-unconditionally` which does the same as
>> `font-lock-mode` but skips the (or noninteractive (eq (aref
>> (buffer-name) 0) ?\s)) test.
>
> Ouch!

I don't understand this reaction.  The code would be just as clean as
the current `font-lock-mode` and it's not like there's any reason to
think that people would abuse such a function.


        Stefan

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> It would be nice if we could somehow enable font-lock inside " *hidden*"
>> buffers. Would it be possible?
>
> Depends on the "somehow" part.  I generally prefer that obscure corner
> cases be solved in the software which creates those obscure cases.  In
> this case, it seems like web-mode is that place?  I'm not sure why
> you say Org should solve this, but perhaps I didn't understand the
> description well enough.  But if this indeed needs to be solved in
> Org, how about not using buffers whose names begin with a space?

While this particular case might be solved on web-mode side (I guess
that directly setting 'font-lock-face property is not recommended?), I
am more concerned about similar issues that might popup in future.

Is it always guaranteed by Emacs that font-lock-fontify-buffer correctly
fontifies that buffer even when font-lock-mode is disabled? If not, Org
has to make sure that font-lock-mode can be enabled when we need to
compute fontification programmatically.

The need to compute fontification is not limited to the described Reddit
report. Org also needs fontification for export purposes - we copy the
Emacs code colours into exported documents. With current behaviour of
font-lock-mode, programmatic fontification may be problematic not only
when we use "hidden" buffers starting with space, but also in
non-interactive mode as in recent related bug report
https://orgmode.org/list/wxaUFiqi8BmIPv8pcYRVHAFa0hTzM35roQxpVVRkgddjRkesPGX1kVBL3G0fr42FonlRq5FNjapV8RiovXV-RGEDehXn-cmIebf4HWBhzIQ=@protonmail.com

As for not using buffers with names starting with a space, we do need
such buffers as means to fontify foreign major mode blocks inside Org.
How else do you suggest computing fontification of an arbitrary text in
arbitrary major-mode (not org-mode) without polluting the buffer list?

Best,
Ihor

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Org mode fontifies code blocks using "hidden" temporary buffers with
>> their name starting from " ": " *org-src-fontification:major-mode*".
>>
>> The buffers are kept around to avoid extra major-mode loading overheads
>> and the fontification is not using (font-lock-ensure).
>
> IIUC they're never displayed, right?  So, `font-lock-ensure` does sound
> like "the right tool".  Could you give a short explanation for why you
> don't use `font-lock-ensure`?

Sorry, it is a typo. I did mean "the fontification *is* using
(font-lock-ensure)". We do use `font-lock-ensure', but it does not
always produce the same results with/without font-lock-mode enabled.

Best,
Ihor

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: yantar92@gmail.com,  56637@debbugs.gnu.org
> Date: Tue, 19 Jul 2022 17:12:44 -0400
> 
> Of course, people can use hacks like
> 
>    (defun really-turn-on-font-lock ()
>      (unwind-protect
>          (let ((noninteractive nil))
>            (rename-buffer (concat "\0" (buffer-name)))
>            (font-lock-mode 1))
>        (when (eq ?\0 (aref (buffer-name) 0))
>          (rename-buffer (substring (buffer-name) 1)))))
> 
> But I can't see why we shouldn't accommodate those needs more directly.

I can: we don't need, and really shouldn't, attempt to cater to each
corner use case in core.  Doing that (and we've been doing that for
quite some time) makes Emacs a larger and less maintainable monster
than it needs to be or already is.  The gain for everyone is minimal,
the gain for those who must for some reason cope with such situations
is small (compared with the above alternative or something like it),
while the damage in terms of being able to know what Emacs does
without stepping through the code with a debugger -- that damage is
quite significant.  You already have one symptom of the monster's
size: I already cannot tell which hooks are and aren't running in
temporary buffers without consulting the sources.  Way to go, Emacs!

> It's not like there a deep technical reason why font-lock should not be
> enabled under any circumstance in those cases.

There are practical reasons why it shouldn't in the vast majority of
cases.  The few rare cases which do have easy workarounds, and I see
absolutely no reason why they couldn't take one of them.

> >> If you don't like `font-lock-allow-in-temporary-buffer`, we could have
> >> a new function `font-lock-enable-unconditionally` which does the same as
> >> `font-lock-mode` but skips the (or noninteractive (eq (aref
> >> (buffer-name) 0) ?\s)) test.
> >
> > Ouch!
> 
> I don't understand this reaction.

See above.

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  56637@debbugs.gnu.org
> Date: Wed, 20 Jul 2022 12:13:14 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> It would be nice if we could somehow enable font-lock inside " *hidden*"
> >> buffers. Would it be possible?
> >
> > Depends on the "somehow" part.  I generally prefer that obscure corner
> > cases be solved in the software which creates those obscure cases.  In
> > this case, it seems like web-mode is that place?  I'm not sure why
> > you say Org should solve this, but perhaps I didn't understand the
> > description well enough.  But if this indeed needs to be solved in
> > Org, how about not using buffers whose names begin with a space?
> 
> While this particular case might be solved on web-mode side (I guess
> that directly setting 'font-lock-face property is not recommended?), I
> am more concerned about similar issues that might popup in future.

We don't necessarily need to solve future problems.  Let's leave
something for those who will come after us.

> Is it always guaranteed by Emacs that font-lock-fontify-buffer correctly
> fontifies that buffer even when font-lock-mode is disabled?

IMO, if it doesn't, it's a bug that should be fixed.  If you have
recipes where it happens, please submit a bug report.

> The need to compute fontification is not limited to the described Reddit
> report. Org also needs fontification for export purposes - we copy the
> Emacs code colours into exported documents. With current behaviour of
> font-lock-mode, programmatic fontification may be problematic not only
> when we use "hidden" buffers starting with space, but also in
> non-interactive mode as in recent related bug report
> https://orgmode.org/list/wxaUFiqi8BmIPv8pcYRVHAFa0hTzM35roQxpVVRkgddjRkesPGX1kVBL3G0fr42FonlRq5FNjapV8RiovXV-RGEDehXn-cmIebf4HWBhzIQ=@protonmail.com

If you create buffer, process text in it, then write it out and kill
the buffer, all in one go, there should be no reason to use a buffer
whose name begins with a space, because interactive users will not
have an opportunity to see such a buffer in the Emacs session.

This problem is only real when a temporary buffer is left around
because, for example, it is being collected/processed asynchronously,
or in several stages with the user being able to interact with Emacs
in-between.

> As for not using buffers with names starting with a space, we do need
> such buffers as means to fontify foreign major mode blocks inside Org.
> How else do you suggest computing fontification of an arbitrary text in
> arbitrary major-mode (not org-mode) without polluting the buffer list?

Once again, if the buffer is killed before the user can see it,
there's no problem to use a name which doesn't start with a space.

But let me turn the table and ask you: how have you been doing this
till now?

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Sorry, it is a typo. I did mean "the fontification *is* using
> (font-lock-ensure)".

Ah, good, thanks.

> We do use `font-lock-ensure', but it does not always produce the same
> results with/without font-lock-mode enabled.

I'm not completely surprised, because I remember when writing
`font-lock-ensure` that there are many corner cases some of which seem
very difficult to fix without actually enabling `font-lock-mode`, but we
should still strive to be "as close as possible", so ... do you have
a bug report with some concrete cases we could try to fix?


        Stefan

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>    (defun really-turn-on-font-lock ()
>>      (unwind-protect
>>          (let ((noninteractive nil))
>>            (rename-buffer (concat "\0" (buffer-name)))
>>            (font-lock-mode 1))
>>        (when (eq ?\0 (aref (buffer-name) 0))
>>          (rename-buffer (substring (buffer-name) 1)))))
>> 
>> But I can't see why we shouldn't accommodate those needs more directly.
>
> I can: we don't need, and really shouldn't, attempt to cater to each
> corner use case in core.  Doing that (and we've been doing that for
> quite some time) makes Emacs a larger and less maintainable monster
> than it needs to be or already is.

I can't see why adding the patch below would make it less maintainable.
It sure seems much cleaner than the horror quoted above (which suffers
from all kinds of corner case misbehaviors).

> You already have one symptom of the monster's size: I already cannot
> tell which hooks are and aren't running in temporary buffers without
> consulting the sources.  Way to go, Emacs!

And for that reason, I think we should try and avoid such special cases
(such the special case of not enabling font-lock in temp buffers), by
replacing them with other measures which give similar results but
without such special casing.

Regarding `inhibit-buffer-hooks`, I wonder if we shouldn't instead
change `with-temp-buffer` so it sets `kill-buffer-hook` (and the other
2 hooks) buffer-locally to nil.
This way, we get pretty much the same end result (in terms of
performance benefits when there are many buffers) yet the hooks get run
normally.

>> >> If you don't like `font-lock-allow-in-temporary-buffer`, we could have
>> >> a new function `font-lock-enable-unconditionally` which does the same as
>> >> `font-lock-mode` but skips the (or noninteractive (eq (aref
>> >> (buffer-name) 0) ?\s)) test.
>> > Ouch!
>> I don't understand this reaction.
> See above.

"Ouch" to me implies something like an ugly code or the introduction of
something that breaks an abstraction or some such.  Having to write the
above quote would qualify, whereas the patch below doesn't seem
to qualify.


        Stefan


diff --git a/lisp/font-core.el b/lisp/font-core.el
index f92d1e38306..dc6aac98828 100644
--- a/lisp/font-core.el
+++ b/lisp/font-core.el
@@ -133,6 +133,12 @@ font-lock-mode
   ;; batch job) or if the buffer is invisible (the name starts with a space).
   (when (or noninteractive (eq (aref (buffer-name) 0) ?\s))
     (setq font-lock-mode nil))
+  (font-lock-mode-unconditionally (not font-lock-mode)))
+
+(defun font-lock-mode-unconditionally (&optional disable)
+  "Enable font-lock unconditionally.
+If DISABLE is non-nil, then disable unconditionally."
+  (setq font-lock-mode (not disable)
   (funcall font-lock-function font-lock-mode)
   ;; Arrange to unfontify this buffer if we change major mode later.
   (if font-lock-mode

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: yantar92@gmail.com,  56637@debbugs.gnu.org
> Date: Wed, 20 Jul 2022 11:15:20 -0400
> 
> >> But I can't see why we shouldn't accommodate those needs more directly.
> >
> > I can: we don't need, and really shouldn't, attempt to cater to each
> > corner use case in core.  Doing that (and we've been doing that for
> > quite some time) makes Emacs a larger and less maintainable monster
> > than it needs to be or already is.
> 
> I can't see why adding the patch below would make it less maintainable.

Well, I explained that.  If you still don't see it, I guess we'll have
to agree to disagree.

> It sure seems much cleaner than the horror quoted above (which suffers
> from all kinds of corner case misbehaviors).

The horror quoted above won't be in Emacs, so it is of no concern to
us.

> > You already have one symptom of the monster's size: I already cannot
> > tell which hooks are and aren't running in temporary buffers without
> > consulting the sources.  Way to go, Emacs!
> 
> And for that reason, I think we should try and avoid such special cases
> (such the special case of not enabling font-lock in temp buffers), by
> replacing them with other measures which give similar results but
> without such special casing.

Temporary buffers _are_ special cases, whether we want it or not.

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> It sure seems much cleaner than the horror quoted above (which suffers
>> from all kinds of corner case misbehaviors).
> The horror quoted above won't be in Emacs, so it is of no concern to us.

I'm pretty sure it will end up in Emacs.


        Stefan

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> Is it always guaranteed by Emacs that font-lock-fontify-buffer correctly
>> fontifies that buffer even when font-lock-mode is disabled?
>
> IMO, if it doesn't, it's a bug that should be fixed.  If you have
> recipes where it happens, please submit a bug report.

Create a file called bug.el with the following text:

(with-current-buffer (get-buffer-create "Test")
  (emacs-lisp-mode)
  (insert "(message \"Foo\")")
  (font-lock-ensure)
  (message "%S" (buffer-string)))

Then, run
emacs-29-vcs -Q --script /tmp/bug.el

Observed output: #("(message \"Foo\")" 9 14 (face font-lock-string-face))
Expected: #("(message \"Foo\")" 0 9 (fontified t) 9 14 (face font-lock-string-face fontified t) 14 15 (fontified t))

The expected output appears in *Messages* when you run 
emacs-29-vcs -Q -l /tmp/bug.el

>> The need to compute fontification is not limited to the described Reddit
>> report. Org also needs fontification for export purposes - we copy the
>> Emacs code colours into exported documents. With current behaviour of
>> font-lock-mode, programmatic fontification may be problematic not only
>> when we use "hidden" buffers starting with space, but also in
>> non-interactive mode as in recent related bug report
>> https://orgmode.org/list/wxaUFiqi8BmIPv8pcYRVHAFa0hTzM35roQxpVVRkgddjRkesPGX1kVBL3G0fr42FonlRq5FNjapV8RiovXV-RGEDehXn-cmIebf4HWBhzIQ=@protonmail.com
>
> If you create buffer, process text in it, then write it out and kill
> the buffer, all in one go, there should be no reason to use a buffer
> whose name begins with a space, because interactive users will not
> have an opportunity to see such a buffer in the Emacs session.
>
> This problem is only real when a temporary buffer is left around
> because, for example, it is being collected/processed asynchronously,
> or in several stages with the user being able to interact with Emacs
> in-between.

Killing the buffer where we do fontification is too slow for Org.
If Org were to use temporary buffers, it would need to load major mode
for every text fragment to be fontified. Loading major modes (at least
some major modes) is taking significant amount of time and should
better be avoided if we can simply reuse a single buffer with major mode
that is already loaded.

>> As for not using buffers with names starting with a space, we do need
>> such buffers as means to fontify foreign major mode blocks inside Org.
>> How else do you suggest computing fontification of an arbitrary text in
>> arbitrary major-mode (not org-mode) without polluting the buffer list?
>
> Once again, if the buffer is killed before the user can see it,
> there's no problem to use a name which doesn't start with a space.
>
> But let me turn the table and ask you: how have you been doing this
> till now?

See the above.

Best,
Ihor

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> We do use `font-lock-ensure', but it does not always produce the same
>> results with/without font-lock-mode enabled.
>
> I'm not completely surprised, because I remember when writing
> `font-lock-ensure` that there are many corner cases some of which seem
> very difficult to fix without actually enabling `font-lock-mode`, but we
> should still strive to be "as close as possible", so ... do you have
> a bug report with some concrete cases we could try to fix?

Apart from the reddit thread I mentioned, where the problem comes from
web-mode directly setting font-lock-face, I also provided an example
with noninteractive Emacs in my reply to Eli.

Best,
Ihor

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: monnier@iro.umontreal.ca,  56637@debbugs.gnu.org
> Date: Thu, 21 Jul 2022 20:09:55 +0800
> 
> >> Is it always guaranteed by Emacs that font-lock-fontify-buffer correctly
> >> fontifies that buffer even when font-lock-mode is disabled?
> >
> > IMO, if it doesn't, it's a bug that should be fixed.  If you have
> > recipes where it happens, please submit a bug report.
> 
> Create a file called bug.el with the following text:
> 
> (with-current-buffer (get-buffer-create "Test")
>   (emacs-lisp-mode)
>   (insert "(message \"Foo\")")
>   (font-lock-ensure)
>   (message "%S" (buffer-string)))
> 
> Then, run
> emacs-29-vcs -Q --script /tmp/bug.el
> 
> Observed output: #("(message \"Foo\")" 9 14 (face font-lock-string-face))
> Expected: #("(message \"Foo\")" 0 9 (fontified t) 9 14 (face font-lock-string-face fontified t) 14 15 (fontified t))
> 
> The expected output appears in *Messages* when you run 
> emacs-29-vcs -Q -l /tmp/bug.el

You said font-lock-fontify-buffer, not font-lock-ensure...

Anyway, I hope Stefan will look into this.

> > If you create buffer, process text in it, then write it out and kill
> > the buffer, all in one go, there should be no reason to use a buffer
> > whose name begins with a space, because interactive users will not
> > have an opportunity to see such a buffer in the Emacs session.
> >
> > This problem is only real when a temporary buffer is left around
> > because, for example, it is being collected/processed asynchronously,
> > or in several stages with the user being able to interact with Emacs
> > in-between.
> 
> Killing the buffer where we do fontification is too slow for Org.
> If Org were to use temporary buffers, it would need to load major mode
> for every text fragment to be fontified. Loading major modes (at least
> some major modes) is taking significant amount of time and should
> better be avoided if we can simply reuse a single buffer with major mode
> that is already loaded.

Sorry, I don't understand: once Emacs loads a major mode, it stays
loaded, unless you forcibly unload it.  Or what do you mean by "load
major mode"?

And why do you assume that erasing a buffer and then inserting some
text into it will be significantly faster than turning on a mode in
it?  It sounds like another fragile assumption.

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> You said font-lock-fontify-buffer, not font-lock-ensure...

Either way. They should both fontify the buffer with font-lock-ensure
not skipping any parts guaranteed.

In any case, changing font-lock-ensure to font-lock-fontify-buffer will
not change failure to fontify using the report steps.

>> Killing the buffer where we do fontification is too slow for Org.
>> If Org were to use temporary buffers, it would need to load major mode
>> for every text fragment to be fontified. Loading major modes (at least
>> some major modes) is taking significant amount of time and should
>> better be avoided if we can simply reuse a single buffer with major mode
>> that is already loaded.
>
> Sorry, I don't understand: once Emacs loads a major mode, it stays
> loaded, unless you forcibly unload it.  Or what do you mean by "load
> major mode"?

If fontification is done in temporary throwaway buffers that are closed
immediately after fontification, next portion of text that should be
fontified in the same major mode will need to create a new throwaway
buffer, turn on the relevant major mode, and perform fontification.
Hence, major mode will need to be loaded every single time we need to
fontify a text fragment.

> And why do you assume that erasing a buffer and then inserting some
> text into it will be significantly faster than turning on a mode in
> it?  It sounds like another fragile assumption.

It is usually true from my experience.

To illustrate, try to create a buffer and run the following:

(progn (emacs-lisp-mode) (benchmark-run 100 (erase-buffer) (insert "(+ 1 2)") (font-lock-ensure)))

(benchmark-run 100 (erase-buffer) (insert "(+ 1 2)") (emacs-lisp-mode) (font-lock-ensure) (fundamental-mode))

The results are:

(5.531764251 0 0.0)
(0.012424528 0 0.0)

Over 400x difference.

Best,
Ihor

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@gmail.com>
> Cc: monnier@iro.umontreal.ca,  56637@debbugs.gnu.org
> Date: Thu, 21 Jul 2022 20:44:23 +0800
> 
> > Sorry, I don't understand: once Emacs loads a major mode, it stays
> > loaded, unless you forcibly unload it.  Or what do you mean by "load
> > major mode"?
> 
> If fontification is done in temporary throwaway buffers that are closed
> immediately after fontification, next portion of text that should be
> fontified in the same major mode will need to create a new throwaway
> buffer, turn on the relevant major mode, and perform fontification.
> Hence, major mode will need to be loaded every single time we need to
> fontify a text fragment.

It is not "loaded", it is "activated".  "Loading" in Emacs means
loading the Lisp package that implements the mode, and that is done
only once.  We don't unload packages when buffers are killed.

> > And why do you assume that erasing a buffer and then inserting some
> > text into it will be significantly faster than turning on a mode in
> > it?  It sounds like another fragile assumption.
> 
> It is usually true from my experience.

Well, "usually" is not a guarantee it will always be so.

Anyway, if this is the issue, we could add another way of marking a
buffer as "invisible for user", one that is not based on the buffer's
name.

> (5.531764251 0 0.0)
> (0.012424528 0 0.0)
> 
> Over 400x difference.

Sorry, but this proves nothing, and I'm sure you know it.

To me, this is just one more fragile assumption on which Org code is
based that is bound to be broken some day.

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> If fontification is done in temporary throwaway buffers that are closed
>> immediately after fontification, next portion of text that should be
>> fontified in the same major mode will need to create a new throwaway
>> buffer, turn on the relevant major mode, and perform fontification.
>> Hence, major mode will need to be loaded every single time we need to
>> fontify a text fragment.
>
> It is not "loaded", it is "activated".  "Loading" in Emacs means
> loading the Lisp package that implements the mode, and that is done
> only once.  We don't unload packages when buffers are killed.

Thanks for the clarification.

>> > And why do you assume that erasing a buffer and then inserting some
>> > text into it will be significantly faster than turning on a mode in
>> > it?  It sounds like another fragile assumption.
>> 
>> It is usually true from my experience.
>
> Well, "usually" is not a guarantee it will always be so.

We do not need such a guarantee. The choice between temporary throwaway
buffers and single major-mode buffer where we insert/remove text is
about speed. Even if reusing the buffer is faster in 90% of major modes
and not 100%, it is a big improvement.

> Anyway, if this is the issue, we could add another way of marking a
> buffer as "invisible for user", one that is not based on the buffer's
> name.
>
>> (5.531764251 0 0.0)
>> (0.012424528 0 0.0)
>> 
>> Over 400x difference.
>
> Sorry, but this proves nothing, and I'm sure you know it.

It was an _illustration_. The decision to reuse buffer was not
arbitrary. Org mode developers and users tested the throwaway buffer
approach, found it too slow and changed to reusing a single major-mode
buffer. The latter is faster in practice.

> To me, this is just one more fragile assumption on which Org code is
> based that is bound to be broken some day.

It is a heuristic. If it is broken, nothing terrible will happen.
Fontification will still work, albeit slower. If the slowdown will be
reported widely, we can always change to the alternative approach.

Also, I am not sure why you are making such a strong claim about "that
is bound to be broken some day". Throwaway buffer will require (1)
creating buffer; (2) activating major-mode; (3) fontification. Reusable
buffer will require (1) Handling major mode's
before/after-change-functions; (2) fontification.

Before/after-change-functions are fast. If not, the relevant major mode
is a terrible mode anyway and should not be used. There is no such
pressure to optimize the activation time, in comparison.

Best,
Ihor

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Observed output: #("(message \"Foo\")" 9 14 (face font-lock-string-face))
> Expected: #("(message \"Foo\")" 0 9 (fontified t) 9 14 (face
> font-lock-string-face fontified t) 14 15 (fontified t))

`fontified` is not a text-property that belongs to font-lock but to
jit-lock, so its presence or absence depends on how font-lock is being
used (it can also be absent if you set `font-lock-support-mode to nil,
for example, and it can also be present even if you don't use font-lock
at all), so I would count this as "an incorrect expectation" rather
than as a bug.

Why do you care about those `fontified` properties?


        Stefan

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Apart from the reddit thread I mentioned, where the problem comes from
> web-mode directly setting font-lock-face,

I didn't see a clear recipe there, tho.  Setting the `font-lock-face`
directly is normal, so it's not clear that it's a problem on the side of
`web-mode`, really.


        Stefan

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Sorry, it is a typo. I did mean "the fontification *is* using
>> (font-lock-ensure)".
>
> Ah, good, thanks.
>
>> We do use `font-lock-ensure', but it does not always produce the same
>> results with/without font-lock-mode enabled.
>
> I'm not completely surprised, because I remember when writing
> `font-lock-ensure` that there are many corner cases some of which seem
> very difficult to fix without actually enabling `font-lock-mode`, but we
> should still strive to be "as close as possible", so ... do you have
> a bug report with some concrete cases we could try to fix?

Would it make sense to make `font-lock-ensure' try harder here, and
(temporarily) force font-lock mode on (even in temporary buffers) if it
isn't already?

I saw that other proposed patch (with `font-lock-mode-unconditionally'),
but I'm not sure that would be as obvious a solution.

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Eli Zaretskii]

> Cc: 56637@debbugs.gnu.org, Ihor Radchenko <yantar92@gmail.com>
> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Sat, 23 Jul 2022 09:53:55 +0200
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> >> Sorry, it is a typo. I did mean "the fontification *is* using
> >> (font-lock-ensure)".
> >
> > Ah, good, thanks.
> >
> >> We do use `font-lock-ensure', but it does not always produce the same
> >> results with/without font-lock-mode enabled.
> >
> > I'm not completely surprised, because I remember when writing
> > `font-lock-ensure` that there are many corner cases some of which seem
> > very difficult to fix without actually enabling `font-lock-mode`, but we
> > should still strive to be "as close as possible", so ... do you have
> > a bug report with some concrete cases we could try to fix?
> 
> Would it make sense to make `font-lock-ensure' try harder here, and
> (temporarily) force font-lock mode on (even in temporary buffers) if it
> isn't already?

That could be a good alternative, yes.  We could even add a new
optional argument for that, if it makes sense to allow controlling
this behavior.

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> I'm not completely surprised, because I remember when writing
>> `font-lock-ensure` that there are many corner cases some of which seem
>> very difficult to fix without actually enabling `font-lock-mode`, but we
>> should still strive to be "as close as possible", so ... do you have
>> a bug report with some concrete cases we could try to fix?
>
> Would it make sense to make `font-lock-ensure' try harder here, and
> (temporarily) force font-lock mode on (even in temporary buffers) if it
> isn't already?

I haven't seen a bug report yet, so I don't know what there is to fix,
so I really can't judge.  But my request for a bug report is indeed
because I hope the problem can be fixed on the side of
`font-lock-ensure`.


        Stefan

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Observed output: #("(message \"Foo\")" 9 14 (face font-lock-string-face))
>> Expected: #("(message \"Foo\")" 0 9 (fontified t) 9 14 (face
>> font-lock-string-face fontified t) 14 15 (fontified t))
>
> `fontified` is not a text-property that belongs to font-lock but to
> jit-lock, so its presence or absence depends on how font-lock is being
> used (it can also be absent if you set `font-lock-support-mode to nil,
> for example, and it can also be present even if you don't use font-lock
> at all), so I would count this as "an incorrect expectation" rather
> than as a bug.
>
> Why do you care about those `fontified` properties?

Oops. You are indeed right. I was trying to convert the report from   https://orgmode.org/list/wxaUFiqi8BmIPv8pcYRVHAFa0hTzM35roQxpVVRkgddjRkesPGX1kVBL3G0fr42FonlRq5FNjapV8RiovXV-RGEDehXn-cmIebf4HWBhzIQ=@protonmail.com
to something simpler.

However, Emacs indeed does the fontification correctly here. The origin
of the problem in the linked message has something to do with htmlize
package, not directly with Emacs.

Please ignore this reproducer.

Best,
Ihor

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 1210 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Apart from the reddit thread I mentioned, where the problem comes from
>> web-mode directly setting font-lock-face,
>
> I didn't see a clear recipe there, tho.  Setting the `font-lock-face`
> directly is normal, so it's not clear that it's a problem on the side of
> `web-mode`, really.

When font-lock-mode is active, char-property-alias-alist contains
'(face font-lock-face). Thus, 'face text property is inheriting the
value of the 'font-lock-face property. It is not the case when
font-lock-mode is disabled. Unless we can forever rely on the
implementation detail that 'font-lock-face is always equivalent to the
'face property, there is no easy way to tell which faces will be applied
to the fontified buffer with enabled font-lock-mode.

Best,
Ihor

On Fri, Jul 22, 2022 at 2:39 AM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> > Apart from the reddit thread I mentioned, where the problem comes from
> > web-mode directly setting font-lock-face,
>
> I didn't see a clear recipe there, tho.  Setting the `font-lock-face`
> directly is normal, so it's not clear that it's a problem on the side of
> `web-mode`, really.
>
>
>         Stefan
>
>

[-- Attachment #2: Type: text/html, Size: 1716 bytes --]

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Ihor Radchenko [2022-07-24 14:47:17] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>> Apart from the reddit thread I mentioned, where the problem comes from
>>> web-mode directly setting font-lock-face,
>>
>> I didn't see a clear recipe there, tho.  Setting the `font-lock-face`
>> directly is normal, so it's not clear that it's a problem on the side of
>> `web-mode`, really.
>
> When font-lock-mode is active, char-property-alias-alist contains
> '(face font-lock-face).

Yes, I know, but it would really help to see a concrete case where that
makes a difference, because there are various ways to "fix" this
problem.  They all have upsides and downsides.

> Unless we can forever rely on the implementation detail that
> 'font-lock-face is always equivalent to the 'face property,

It's not really "equivalent", but more importantly it's not an
implementation detail: it's documented in the ELisp manual.


        Stefan

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> When font-lock-mode is active, char-property-alias-alist contains
>> '(face font-lock-face).
>
> Yes, I know, but it would really help to see a concrete case where that
> makes a difference, because there are various ways to "fix" this
> problem.  They all have upsides and downsides.

In my case, the main difference is in org-src-font-lock-fontify-block.
Its purpose is copying the text fontification from major mode buffer to
Org mode buffer. The copying is done by checking 'face and
font-lock-extra-managed-props properties, which is sufficient as long as
font-lock-mode is activated. When font-lock-mode is not activated,
copying 'face is not enough, and we have to copy 'font-lock-face.

>> Unless we can forever rely on the implementation detail that
>> 'font-lock-face is always equivalent to the 'face property,
> It's not really "equivalent", but more importantly it's not an
> implementation detail: it's documented in the ELisp manual.

Apart from 'font-lock-face, which can be copied explicitly, I do not see
any real issues.

Maybe only htmlize, which explicitly checks the value of font-lock-mode
variable and does something strange if it is nil.

Of course, htmlize is an external package. However, potentially, places
that check font-lock-mode value and can be used noninteractively may be
problematic:

1. align-rules-list
2. faces--attribute-at-point
3. font-lock-flush (it will fail to work when font-lock-mode is nil)
4. Info-fontify-node
5. c-toggle-comment-style
6. c-fontify-new-found-type
7. c-change-expand-fl-region
8. grep-process-setup
9. sh-set-shell
10. verilog-preprocess
11. reftex-show-entry

Best,
Ihor

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> In my case, the main difference is in org-src-font-lock-fontify-block.
> Its purpose is copying the text fontification from major mode buffer to
> Org mode buffer. The copying is done by checking 'face and
> font-lock-extra-managed-props properties, which is sufficient as long as
> font-lock-mode is activated. When font-lock-mode is not activated,
> copying 'face is not enough, and we have to copy 'font-lock-face.

You're slowly inching toward a concrete case, but it would be much
simpler and faster to just give a recipe telling what you do, what you
see, and what you'd like to see instead.
You know: a normal bug report.


        Stefan

bug#56637: 28.1.90; [FR] Allow a way around font-lock-mode being unconditionally disabled in " *hidden*" buffers

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> In my case, the main difference is in org-src-font-lock-fontify-block.
>> Its purpose is copying the text fontification from major mode buffer to
>> Org mode buffer. The copying is done by checking 'face and
>> font-lock-extra-managed-props properties, which is sufficient as long as
>> font-lock-mode is activated. When font-lock-mode is not activated,
>> copying 'face is not enough, and we have to copy 'font-lock-face.
>
> You're slowly inching toward a concrete case, but it would be much
> simpler and faster to just give a recipe telling what you do, what you
> see, and what you'd like to see instead.
> You know: a normal bug report.

I am sorry, but there is no useful bug to report here.

I had an actual bug report along the lines:

1. Install web-mode
2. Configure Org mode to use web-mode for html
3. Create html src block in Org mode
5. Fontification does not work

The issue was disabled font-lock-mode in temporary buffer + web-mode
directly setting font-lock-face.

I have fixed this on Org side by explicitly copying font-lock-face
property instead of relying on font-lock-face being remapped to face.

Then, I posted the bug to emacs-devel because (1) I was not sure if
copying face-lock-face is a reliable option (it is, as we figured); (2)
Similar issues might popup as a result of font-lock-mode being disabled
in temporary buffers/noninteractive mode.

Best,
Ihor