bug#68849: 30.0.50; font-lock does not flush fontification after calling open-line

bug#68849: 30.0.50; font-lock does not flush fontification after calling open-line

[Ihor Radchenko]

This bug was originally reported in
https://orgmode.org/list/c5fb9f63-185c-4e36-b13f-e887a4a29aae@yandex.com

I am not 100% sure if is Emacs bug, but my preliminary testing showed
anomalies in `jit-lock-after-change' -
(put-text-property jit-lock-start jit-lock-end 'fontified nil) somehow
did not apply the text properties. I suspect that I missed something
though.

May someone please help debugging this and figuring out if the problem
is on Emacs side?

Steps to reproduce:

1. Create bug.el:
(require 'org-faces)
(set-face-underline 'org-block-end-line t)
(setq org-fontify-whole-block-delimiter-line t)

2. Create bug.org:
#+begin_src
#+end_src

asd

3. Move point to the end of #+end_src line:
#+end_src<point>

4. M-x open-lien

5. Observe underline face being applied to the empty line _after_
#+end_src.

In GNU Emacs 30.0.50 (build 4, x86_64-pc-linux-gnu, GTK+ Version
 3.24.39, cairo version 1.18.0) of 2024-01-30 built on localhost
Repository revision: 9de29fb5b0396cb3929bd5668604fda076ca5ec4
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12101011
System Description: Gentoo Linux

Configured using:
 'configure JAVAC=/etc/java-config-2/current-system-vm/bin/javac'

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#68849: 30.0.50; font-lock does not flush fontification after calling open-line

[Eli Zaretskii]

> Cc: Kostas Papadakis <papadakis.k@yandex.com>
> From: Ihor Radchenko <yantar92@posteo.net>
> Date: Wed, 31 Jan 2024 12:44:09 +0000
> 
> This bug was originally reported in
> https://orgmode.org/list/c5fb9f63-185c-4e36-b13f-e887a4a29aae@yandex.com
> 
> I am not 100% sure if is Emacs bug, but my preliminary testing showed
> anomalies in `jit-lock-after-change' -
> (put-text-property jit-lock-start jit-lock-end 'fontified nil) somehow
> did not apply the text properties. I suspect that I missed something
> though.
> 
> May someone please help debugging this and figuring out if the problem
> is on Emacs side?
> 
> Steps to reproduce:
> 
> 1. Create bug.el:
> (require 'org-faces)
> (set-face-underline 'org-block-end-line t)
> (setq org-fontify-whole-block-delimiter-line t)
> 
> 2. Create bug.org:
> #+begin_src
> #+end_src
> 
> asd
> 
> 3. Move point to the end of #+end_src line:
> #+end_src<point>
> 
> 4. M-x open-lien
> 
> 5. Observe underline face being applied to the empty line _after_
> #+end_src.

Stefan, any suggestions?

bug#68849: 30.0.50; font-lock does not flush fontification after calling open-line

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I am not 100% sure if is Emacs bug, but my preliminary testing showed
> anomalies in `jit-lock-after-change' -
> (put-text-property jit-lock-start jit-lock-end 'fontified nil) somehow
> did not apply the text properties. I suspect that I missed something
> though.

AFAICT the `put-text-property` works fine :-)
The problem is that your `org-fontify-extend-region` does not spread the
change to the "next line".

When `font-lock-extend-after-change-region-function` is nil, Emacs does
that for you.  Quoting `font-lock-extend-jit-lock-region-after-change`:

      (when (< end (point-max))
        (setq end
              (cond
               ((get-text-property end 'font-lock-multiline)
                (or (text-property-any end (point-max)
                                       'font-lock-multiline nil)
                    (point-max)))
               ;; If `end' has been set by the function above, don't corrupt it.
               (font-lock-extend-after-change-region-function end)
                ;; Rounding up to a whole number of lines should include the
                ;; line right after `end'.  Typical case: the first char of
                ;; the line was deleted.  Or a \n was inserted in the middle
                ;; of a line.
               (t (1+ end)))))

I still hope you can change Org mode so it uses
`font-lock-extend-region-functions` instead of
`font-lock-extend-after-change-region-function`.


        Stefan

bug#68849: 30.0.50; font-lock does not flush fontification after calling open-line

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I am not 100% sure if is Emacs bug, but my preliminary testing showed
>> anomalies in `jit-lock-after-change' -
>> (put-text-property jit-lock-start jit-lock-end 'fontified nil) somehow
>> did not apply the text properties. I suspect that I missed something
>> though.
>
> AFAICT the `put-text-property` works fine :-)
> The problem is that your `org-fontify-extend-region` does not spread the
> change to the "next line".
>
> When `font-lock-extend-after-change-region-function` is nil, Emacs does
> that for you.  Quoting `font-lock-extend-jit-lock-region-after-change`:
> ...

Thanks for the pointer!
Fixed, on bugfix.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=1abff3859

> I still hope you can change Org mode so it uses
> `font-lock-extend-region-functions` instead of
> `font-lock-extend-after-change-region-function`.

I am not taking on any major changes in the existing Org fontification
as it needs to be rewritten completely.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#68849: 30.0.50; font-lock does not flush fontification after calling open-line

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> I still hope you can change Org mode so it uses
>> `font-lock-extend-region-functions` instead of
>> `font-lock-extend-after-change-region-function`.
>
> I am not taking on any major changes in the existing Org fontification
> as it needs to be rewritten completely.

So I heard, yes.  You won't crush my hopes so easily.
Oh, and BTW, in order to try and prevent that famous rewrite from making
the same mistake, a FIXME would be appreciated.


        Stefan