bug#22399: org-mode fontification

bug#22399: org-mode fontification

[Phillip Lord]

On emacs-25 ox-html, line 1847, org-html-fontify-code does this...


(setq code (with-temp-buffer
		       ;; Switch to language-specific mode.
		       (funcall lang-mode)
		       (insert code)
		       ;; Fontify buffer.
		       (org-font-lock-ensure)


But `org-font-lock-ensure' is defined like so....

(defalias 'org-font-lock-ensure
  (if (fboundp 'org-font-lock-ensure)
      #'font-lock-ensure
    (lambda (_beg _end) (font-lock-fontify-buffer))))


which takes two args, causing crashes when exporting to code to HTML.

Phil

bug#22399: org-mode fontification

[Rasmus]

Hi Phillip,

Thanks for your report.

phillip.lord@russet.org.uk (Phillip Lord) writes:

> On emacs-25 ox-html, line 1847, org-html-fontify-code does this...
>
> (setq code (with-temp-buffer
> 		       ;; Switch to language-specific mode.
> 		       (funcall lang-mode)
> 		       (insert code)
> 		       ;; Fontify buffer.
> 		       (org-font-lock-ensure)
>
>
> But `org-font-lock-ensure' is defined like so....
>
> (defalias 'org-font-lock-ensure
>   (if (fboundp 'org-font-lock-ensure)
>       #'font-lock-ensure
>     (lambda (_beg _end) (font-lock-fontify-buffer))))
>
>
> which takes two args, causing crashes when exporting to code to HTML.

This is fixed in the most recent stable version of Org, i.e. 8.3.3¹ ,
which, I guess, will be part of if Emacs 25.2 or 26.1 (whichever comes
first and allow new features).  It’s also fixed in 8.2.10².  I guess
Emacs-25 should switch v8.2.10.  [Honestly, when v8.3 couldn’t be merged I
did not cross my mind to upgrade to v8.2.10].  In case v8.2.10 it too
radical, a "local fix" must be applied to the version of Org in emacs.git.

Thanks,
Rasmus


Footnotes: 
¹   http://orgmode.org/cgit.cgi/org-mode.git/tree/lisp/ox-html.el?h=maint#n2086

²   http://orgmode.org/cgit.cgi/org-mode.git/tree/lisp/ox-html.el?id=fdd9b18598239dd0e8b18383ac930110d51ed22e#n1893

-- 
Warning: Everything saved will be lost

bug#22399: org-mode fontification

[Phillip Lord]

Rasmus <rasmus@gmx.us> writes:

> Hi Phillip,
>
> Thanks for your report.
>
> phillip.lord@russet.org.uk (Phillip Lord) writes:
>
>> On emacs-25 ox-html, line 1847, org-html-fontify-code does this...
>>
>> (setq code (with-temp-buffer
>> 		       ;; Switch to language-specific mode.
>> 		       (funcall lang-mode)
>> 		       (insert code)
>> 		       ;; Fontify buffer.
>> 		       (org-font-lock-ensure)
>>
>>
>> But `org-font-lock-ensure' is defined like so....
>>
>> (defalias 'org-font-lock-ensure
>>   (if (fboundp 'org-font-lock-ensure)
>>       #'font-lock-ensure
>>     (lambda (_beg _end) (font-lock-fontify-buffer))))
>>
>>
>> which takes two args, causing crashes when exporting to code to HTML.
>
> This is fixed in the most recent stable version of Org, i.e. 8.3.3¹ ,
> which, I guess, will be part of if Emacs 25.2 or 26.1 (whichever comes
> first and allow new features).  It’s also fixed in 8.2.10².  I guess
> Emacs-25 should switch v8.2.10.  [Honestly, when v8.3 couldn’t be merged I
> did not cross my mind to upgrade to v8.2.10].  In case v8.2.10 it too
> radical, a "local fix" must be applied to the version of Org in emacs.git.


Well, the fix on emacs-25 appears to be trivial i.e.

(org-font-lock-ensure (point-min) (point-max))

But I don't know what the protocol is for org-mode as it's upstream.
AFAICT, this is a regression from emacs-24.5, so it would be a shame if
the bug gets into 25.1.

Let me know if you want me to do anything!

Phil

bug#22399: org-mode fontification

[Rasmus]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> Rasmus <rasmus@gmx.us> writes:
>
>> Hi Phillip,
>>
>> Thanks for your report.
>>
>> phillip.lord@russet.org.uk (Phillip Lord) writes:
>>
>>> On emacs-25 ox-html, line 1847, org-html-fontify-code does this...
>>>
>>> (setq code (with-temp-buffer
>>> 		       ;; Switch to language-specific mode.
>>> 		       (funcall lang-mode)
>>> 		       (insert code)
>>> 		       ;; Fontify buffer.
>>> 		       (org-font-lock-ensure)
>>>
>>>
>>> But `org-font-lock-ensure' is defined like so....
>>>
>>> (defalias 'org-font-lock-ensure
>>>   (if (fboundp 'org-font-lock-ensure)
>>>       #'font-lock-ensure
>>>     (lambda (_beg _end) (font-lock-fontify-buffer))))
>>>
>>>
>>> which takes two args, causing crashes when exporting to code to HTML.
>>
>> This is fixed in the most recent stable version of Org, i.e. 8.3.3¹ ,
>> which, I guess, will be part of if Emacs 25.2 or 26.1 (whichever comes
>> first and allow new features).  It’s also fixed in 8.2.10².  I guess
>> Emacs-25 should switch v8.2.10.  [Honestly, when v8.3 couldn’t be merged I
>> did not cross my mind to upgrade to v8.2.10].  In case v8.2.10 it too
>> radical, a "local fix" must be applied to the version of Org in emacs.git.
>
>
> Well, the fix on emacs-25 appears to be trivial i.e.
>
> (org-font-lock-ensure (point-min) (point-max))
>
> But I don't know what the protocol is for org-mode as it's upstream.
> AFAICT, this is a regression from emacs-24.5, so it would be a shame if
> the bug gets into 25.1.

Sorry, I managed to confuse myself before.  Emacs-25 has Org 8.2.10, but
this bug only lives in the core version.

Isn’t a better fix to change the signature of the alias to mirror that of
font-lock-ensure?  I.e.

    (defalias 'org-font-lock-ensure
      (if (fboundp 'org-font-lock-ensure)
          #'font-lock-ensure
        (lambda (&optional _beg _end) (font-lock-fontify-buffer))))

Since (org-font-lock-ensure) is called from many places.

Note, this change only lives it the Emacs version of Org.  It was added in
this commit:

       6711a21f1125c0047c56eb266eb374c1ec90a967
       Author:     Stefan Monnier <monnier@iro.umontreal.ca>
       AuthorDate: Wed May 28 23:45:29 2014 -0400

      * lisp/font-lock.el (font-lock-flush, font-lock-ensure): New functions.
      * lisp/org/org-compat.el (org-font-lock-ensure): New function.
      * lisp/org/ox-odt.el (org-odt-do-format-code):
      * lisp/org/ox-html.el (org-html-fontify-code):
      * lisp/org/org.el (org-fontify-like-in-org-mode):
      * lisp/org/org-src.el (org-src-font-lock-fontify-block):
      * lisp/org/org-clock.el (org-clock-get-clocktable): Use it.
      * lisp/org/ox-org.el (org-org-publish-to-org): Use it.  Avoid using find-file
      from Elisp.
      * test/automated/ruby-mode-tests.el (ruby-assert-face): Use font-lock-ensure.
      (ruby-interpolation-keeps-non-quote-syntax): Use syntax-propertize.


I’ll check on font-lock-ensure business and whether we can use that in
Org-9.

Thanks,
Rasmus

-- 
The right to be left alone is a human right

bug#22399: org-mode fontification

[Kyle Meyer]

Hi Phillip and Rasmus,

Rasmus <rasmus@gmx.us> writes:

[...]

> Note, this change only lives it the Emacs version of Org.  It was added in
> this commit:
>
>        6711a21f1125c0047c56eb266eb374c1ec90a967
>        Author:     Stefan Monnier <monnier@iro.umontreal.ca>
>        AuthorDate: Wed May 28 23:45:29 2014 -0400
>
>       * lisp/font-lock.el (font-lock-flush, font-lock-ensure): New functions.
>       * lisp/org/org-compat.el (org-font-lock-ensure): New function.
>       * lisp/org/ox-odt.el (org-odt-do-format-code):
>       * lisp/org/ox-html.el (org-html-fontify-code):
>       * lisp/org/org.el (org-fontify-like-in-org-mode):
>       * lisp/org/org-src.el (org-src-font-lock-fontify-block):
>       * lisp/org/org-clock.el (org-clock-get-clocktable): Use it.
>       * lisp/org/ox-org.el (org-org-publish-to-org): Use it.  Avoid using find-file
>       from Elisp.
>       * test/automated/ruby-mode-tests.el (ruby-assert-face): Use font-lock-ensure.
>       (ruby-interpolation-keeps-non-quote-syntax): Use syntax-propertize.

This commit was backported in Org commit d81e6b5, and then followed up
by c115c7b (org-compat: Use different font-lock-ensure alias,
2015-07-26), which defined the alias as

    (unless (fboundp 'font-lock-ensure)
      (defalias 'font-lock-ensure 'font-lock-fontify-buffer))


--
Kyle

bug#22399: org-mode fontification

[Kyle Meyer]

[-- Attachment #1: Type: text/plain, Size: 1214 bytes --]

Kyle Meyer <kyle@kyleam.com> writes:
> Rasmus <rasmus@gmx.us> writes:
>
> [...]
>
>> Note, this change only lives it the Emacs version of Org.  It was added in
>> this commit:
>>
>>        6711a21f1125c0047c56eb266eb374c1ec90a967

[...]

>
> This commit was backported in Org commit d81e6b5, and then followed up
> by c115c7b (org-compat: Use different font-lock-ensure alias,
> 2015-07-26), which defined the alias as
>
>     (unless (fboundp 'font-lock-ensure)
>       (defalias 'font-lock-ensure 'font-lock-fontify-buffer))

I've attached two patches against emacs-25 that would get the Org and
Emacs repos in similar states wrt font-lock-ensure.

The first patch replaces the font-lock-ensure compatibility alias from
6711a21f1 with the one currently used in the Org repo (from commit
e6883dd03).  The new alias is limited to being called with no
arguments, but, at the moment, no Org code calls font-lock-ensure with
any arguments.

The second patch reverts a part of 6711a21f1 that modified
org-src-font-lock-fontify-block to use font-lock-ensure rather than
font-lock-fontify-buffer.  In this particular case, using
font-lock-ensure instead of font-lock-fontify-buffer fails to
highlight the source block.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-compat-Use-different-font-lock-ensure-alias.patch --]
[-- Type: text/x-diff, Size: 4525 bytes --]

From 6072152a45bcdebbadbe7ec8f7589fce4bc341f2 Mon Sep 17 00:00:00 2001
From: Kyle Meyer <kyle@kyleam.com>
Date: Sun, 26 Jul 2015 23:29:14 -0400
Subject: [PATCH 1/2] org-compat: Use different font-lock-ensure alias

* lisp/org/org-compat.el (org-font-lock-ensure): Remove in favor of
font-lock-ensure alias from Org's e6883dd.  (Bug#22399)
* lisp/org/org-clock.el (org-clock-get-clocktable):
* lisp/org/org-src.el (org-src-font-lock-fontify-block):
* lisp/org/org.el (org-fontify-like-in-org-mode):
* lisp/org/ox-html.el (org-html-fontify-code):
* lisp/org/ox-odt.el (org-odt-do-format-code):
* lisp/org/ox-org.el (org-org-publish-to-org): Use new alias.

Instead of org-font-lock-ensure alias introduced in 6711a21, use
font-lock-ensure alias added in Org's e6883dd ("org-compat: Provide
compatibility definition for font-lock-ensure", 2014-06-01).

This corresponds to c115c7b4e86b68fe4f903636252a8b22b46a044f in the Org
repo.
---
 lisp/org/org-clock.el  | 2 +-
 lisp/org/org-compat.el | 7 +++----
 lisp/org/org-src.el    | 2 +-
 lisp/org/org.el        | 2 +-
 lisp/org/ox-html.el    | 2 +-
 lisp/org/ox-odt.el     | 2 +-
 lisp/org/ox-org.el     | 2 +-
 7 files changed, 9 insertions(+), 10 deletions(-)

diff --git a/lisp/org/org-clock.el b/lisp/org/org-clock.el
index 9374f5f..217e97e 100644
--- a/lisp/org/org-clock.el
+++ b/lisp/org/org-clock.el
@@ -1931,7 +1931,7 @@ org-clock-get-clocktable
     (org-mode)
     (org-create-dblock props)
     (org-update-dblock)
-    (org-font-lock-ensure)
+    (font-lock-ensure)
     (forward-line 2)
     (buffer-substring (point) (progn
 				(re-search-forward "^[ \t]*#\\+END" nil t)
diff --git a/lisp/org/org-compat.el b/lisp/org/org-compat.el
index 6403d4d..863197b 100644
--- a/lisp/org/org-compat.el
+++ b/lisp/org/org-compat.el
@@ -478,10 +478,9 @@ org-no-popups
 	      (looking-at (concat "\\(?:"  regexp "\\)\\'")))))
       (not (null pos)))))
 
-(defalias 'org-font-lock-ensure
-  (if (fboundp 'org-font-lock-ensure)
-      #'font-lock-ensure
-    (lambda (_beg _end) (font-lock-fontify-buffer))))
+;; `font-lock-ensure' is only available from 24.4.50 on
+(unless (fboundp 'font-lock-ensure)
+  (defalias 'font-lock-ensure 'font-lock-fontify-buffer))
 
 (defun org-floor* (x &optional y)
   "Return a list of the floor of X and the fractional part of X.
diff --git a/lisp/org/org-src.el b/lisp/org/org-src.el
index 8e06aaa..1bcc202 100644
--- a/lisp/org/org-src.el
+++ b/lisp/org/org-src.el
@@ -916,7 +916,7 @@ org-src-font-lock-fontify-block
 	    (delete-region (point-min) (point-max))
 	    (insert string " ") ;; so there's a final property change
 	    (unless (eq major-mode lang-mode) (funcall lang-mode))
-            (org-font-lock-ensure)
+	    (font-lock-ensure)
 	    (setq pos (point-min))
 	    (while (setq next (next-single-property-change pos 'face))
 	      (put-text-property
diff --git a/lisp/org/org.el b/lisp/org/org.el
index 960b874..bb3a89e 100644
--- a/lisp/org/org.el
+++ b/lisp/org/org.el
@@ -6375,7 +6375,7 @@ org-fontify-like-in-org-mode
     (insert s)
     (let ((org-odd-levels-only odd-levels))
       (org-mode)
-      (org-font-lock-ensure)
+      (font-lock-ensure)
       (buffer-string))))
 
 (defvar org-m nil)
diff --git a/lisp/org/ox-html.el b/lisp/org/ox-html.el
index 1ca360c..b229423 100644
--- a/lisp/org/ox-html.el
+++ b/lisp/org/ox-html.el
@@ -1855,7 +1855,7 @@ org-html-fontify-code
 		       (funcall lang-mode)
 		       (insert code)
 		       ;; Fontify buffer.
-		       (org-font-lock-ensure)
+		       (font-lock-ensure)
 		       ;; Remove formatting on newline characters.
 		       (save-excursion
 			 (let ((beg (point-min))
diff --git a/lisp/org/ox-odt.el b/lisp/org/ox-odt.el
index f2b0c91..4f9760d 100644
--- a/lisp/org/ox-odt.el
+++ b/lisp/org/ox-odt.el
@@ -3140,7 +3140,7 @@ org-odt-do-format-code
 		 (with-temp-buffer
 		   (insert code)
 		   (funcall lang-mode)
-		   (org-font-lock-ensure)
+		   (font-lock-ensure)
 		   (buffer-string))))
 	 (fontifier (if use-htmlfontify-p 'org-odt-htmlfontify-string
 		      'org-odt--encode-plain-text))
diff --git a/lisp/org/ox-org.el b/lisp/org/ox-org.el
index 6221c70..bc4eff9 100644
--- a/lisp/org/ox-org.el
+++ b/lisp/org/ox-org.el
@@ -254,7 +254,7 @@ org-org-publish-to-org
 	   (work-buffer (or visitingp (find-file-noselect filename)))
 	   newbuf)
       (with-current-buffer work-buffer
-        (org-font-lock-ensure)
+        (font-lock-ensure)
         (show-all)
         (org-show-block-all)
         (setq newbuf (htmlize-buffer)))
-- 
2.7.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-org-src-Use-font-lock-fontify-buffer-for-blocks.patch --]
[-- Type: text/x-diff, Size: 1164 bytes --]

From 212c3131def8d765e7ae4698f931530033cb9144 Mon Sep 17 00:00:00 2001
From: Kyle Meyer <kyle@kyleam.com>
Date: Tue, 4 Aug 2015 23:50:59 -0400
Subject: [PATCH 2/2] org-src: Use font-lock-fontify-buffer for blocks

* lisp/org/org-src.el (org-src-font-lock-fontify-block): Use
font-lock-fontify-buffer because font-lock-ensure does not display
source block fonts.

This corresponds to 003a0f10695f035e844d844eacb1a86a6d2df934 in the Org
repo.
---
 lisp/org/org-src.el | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lisp/org/org-src.el b/lisp/org/org-src.el
index 1bcc202..5548f52 100644
--- a/lisp/org/org-src.el
+++ b/lisp/org/org-src.el
@@ -916,7 +916,9 @@ org-src-font-lock-fontify-block
 	    (delete-region (point-min) (point-max))
 	    (insert string " ") ;; so there's a final property change
 	    (unless (eq major-mode lang-mode) (funcall lang-mode))
-	    (font-lock-ensure)
+	    ;; Avoid `font-lock-ensure', which does not display fonts in
+	    ;; source block.
+	    (font-lock-fontify-buffer)
 	    (setq pos (point-min))
 	    (while (setq next (next-single-property-change pos 'face))
 	      (put-text-property
-- 
2.7.0


[-- Attachment #4: Type: text/plain, Size: 9 bytes --]


--
Kyle

bug#22399: org-mode fontification

[Phillip Lord]

Kyle Meyer <kyle@kyleam.com> writes:

> Kyle Meyer <kyle@kyleam.com> writes:
>> Rasmus <rasmus@gmx.us> writes:
>>
>> [...]
>>
>>> Note, this change only lives it the Emacs version of Org.  It was added in
>>> this commit:
>>>
>>>        6711a21f1125c0047c56eb266eb374c1ec90a967
>
> [...]
>
>>
>> This commit was backported in Org commit d81e6b5, and then followed up
>> by c115c7b (org-compat: Use different font-lock-ensure alias,
>> 2015-07-26), which defined the alias as
>>
>>     (unless (fboundp 'font-lock-ensure)
>>       (defalias 'font-lock-ensure 'font-lock-fontify-buffer))
>
> I've attached two patches against emacs-25 that would get the Org and
> Emacs repos in similar states wrt font-lock-ensure.
>
> The first patch replaces the font-lock-ensure compatibility alias from
> 6711a21f1 with the one currently used in the Org repo (from commit
> e6883dd03).  The new alias is limited to being called with no
> arguments, but, at the moment, no Org code calls font-lock-ensure with
> any arguments.
>
> The second patch reverts a part of 6711a21f1 that modified
> org-src-font-lock-fontify-block to use font-lock-ensure rather than
> font-lock-fontify-buffer.  In this particular case, using
> font-lock-ensure instead of font-lock-fontify-buffer fails to
> highlight the source block.


I've tested these on emacs-25 and can confirm that they solve the
problem.

Thanks!

Phil

bug#22399: org-mode fontification

[Rasmus]

The following message is a courtesy copy of an article
that has been posted to gmane.emacs.orgmode as well.

Kyle Meyer <kyle@kyleam.com> writes:

> I've attached two patches against emacs-25 that would get the Org and
> Emacs repos in similar states wrt font-lock-ensure.

Thanks Kyle.

> The first patch replaces the font-lock-ensure compatibility alias from
> 6711a21f1 with the one currently used in the Org repo (from commit
> e6883dd03).  The new alias is limited to being called with no
> arguments, but, at the moment, no Org code calls font-lock-ensure with
> any arguments.

I just have one concern, which is pretty theoretical.  Isn’t it bad to
define the alias to be font-lock-ensure since this is an existing function
and people would have clear expectations of what they are calling?  I
realize that there’s no Emacs-25 that will not have this function (which
is why it’s very theoretical).  Also, if the alias were ever to point to
font-lock-fontify-buffer, the signature would change compared to the
expected signature of font-lock-ensure.

These considerations are probably too theoretical to consider further,
though.

> The second patch reverts a part of 6711a21f1 that modified
> org-src-font-lock-fontify-block to use font-lock-ensure rather than
> font-lock-fontify-buffer.  In this particular case, using
> font-lock-ensure instead of font-lock-fontify-buffer fails to
> highlight the source block.

Cool!

Cheers,
Rasmus

-- 
May contains speling mistake

bug#22399: org-mode fontification

[Kyle Meyer]

Rasmus <rasmus@gmx.us> writes:

> Kyle Meyer <kyle@kyleam.com> writes:

[...]

>> The first patch replaces the font-lock-ensure compatibility alias from
>> 6711a21f1 with the one currently used in the Org repo (from commit
>> e6883dd03).  The new alias is limited to being called with no
>> arguments, but, at the moment, no Org code calls font-lock-ensure with
>> any arguments.
>
> I just have one concern, which is pretty theoretical.  Isn’t it bad to
> define the alias to be font-lock-ensure since this is an existing function
> and people would have clear expectations of what they are calling?  I
> realize that there’s no Emacs-25 that will not have this function (which
> is why it’s very theoretical).

I agree, but, like you, I'm not sure it will matter in practice.  Again,
this was the solution that was already in Org's repo at the time I
backported Emacs's 6711a21f1.  However, it's easy enough to switch to
using an org-font-lock-ensure variant that avoids these issues, so I
will do that (in an updated patch and on Org's maint).

--
Kyle

bug#22399: org-mode fontification

[Kyle Meyer]

[-- Attachment #1: Type: text/plain, Size: 366 bytes --]

Kyle Meyer <kyle@kyleam.com> writes:

[...]

> However, it's easy enough to switch to using an org-font-lock-ensure
> variant that avoids these issues, so I will do that (in an updated
> patch and on Org's maint).

I've attached updated patches against emacs-25.  If these are applied to
the Emacs repo, I'll backport the new font-lock-ensure alias to Org's
maint.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-compat-Fix-font-lock-ensure-alias.patch --]
[-- Type: text/x-patch, Size: 1049 bytes --]

From d9a8014e3919c328bf2b6acbe9d9127f3b06b5e1 Mon Sep 17 00:00:00 2001
From: Kyle Meyer <kyle@kyleam.com>
Date: Tue, 19 Jan 2016 12:53:04 -0500
Subject: [PATCH 1/2] org-compat: Fix font-lock-ensure alias

* lisp/org/org-compat.el (org-font-lock-ensure): Fix check for
font-lock-ensure.  Make compatibility function's arguments optional to
match with font-lock-ensure's.  (Bug#22399)
---
 lisp/org/org-compat.el | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lisp/org/org-compat.el b/lisp/org/org-compat.el
index 6403d4d..cf6aafc 100644
--- a/lisp/org/org-compat.el
+++ b/lisp/org/org-compat.el
@@ -479,9 +479,9 @@ org-no-popups
       (not (null pos)))))
 
 (defalias 'org-font-lock-ensure
-  (if (fboundp 'org-font-lock-ensure)
+  (if (fboundp 'font-lock-ensure)
       #'font-lock-ensure
-    (lambda (_beg _end) (font-lock-fontify-buffer))))
+    (lambda (&optional _beg _end) (font-lock-fontify-buffer))))
 
 (defun org-floor* (x &optional y)
   "Return a list of the floor of X and the fractional part of X.
-- 
2.7.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-org-src-Use-font-lock-fontify-buffer-for-blocks.patch --]
[-- Type: text/x-patch, Size: 1176 bytes --]

From 7b2e8cd23ee5326ca2b0845bed8a5d111a2a3b17 Mon Sep 17 00:00:00 2001
From: Kyle Meyer <kyle@kyleam.com>
Date: Tue, 19 Jan 2016 12:53:22 -0500
Subject: [PATCH 2/2] org-src: Use font-lock-fontify-buffer for blocks

* lisp/org/org-src.el (org-src-font-lock-fontify-block): Use
font-lock-fontify-buffer because font-lock-ensure does not display
source block fonts.

This corresponds to 003a0f10695f035e844d844eacb1a86a6d2df934 in the Org
repo.
---
 lisp/org/org-src.el | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lisp/org/org-src.el b/lisp/org/org-src.el
index 8e06aaa..5548f52 100644
--- a/lisp/org/org-src.el
+++ b/lisp/org/org-src.el
@@ -916,7 +916,9 @@ org-src-font-lock-fontify-block
 	    (delete-region (point-min) (point-max))
 	    (insert string " ") ;; so there's a final property change
 	    (unless (eq major-mode lang-mode) (funcall lang-mode))
-            (org-font-lock-ensure)
+	    ;; Avoid `font-lock-ensure', which does not display fonts in
+	    ;; source block.
+	    (font-lock-fontify-buffer)
 	    (setq pos (point-min))
 	    (while (setq next (next-single-property-change pos 'face))
 	      (put-text-property
-- 
2.7.0


[-- Attachment #4: Type: text/plain, Size: 9 bytes --]


--
Kyle

bug#22399: org-mode fontification

[Rasmus]

The following message is a courtesy copy of an article
that has been posted to gmane.emacs.orgmode as well.

Hi Kyle,

Kyle Meyer <kyle@kyleam.com> writes:

>> I just have one concern, which is pretty theoretical.  Isn’t it bad to
>> define the alias to be font-lock-ensure since this is an existing function
>> and people would have clear expectations of what they are calling?  I
>> realize that there’s no Emacs-25 that will not have this function (which
>> is why it’s very theoretical).
>
> I agree, but, like you, I'm not sure it will matter in practice.  Again,
> this was the solution that was already in Org's repo at the time I
> backported Emacs's 6711a21f1.  However, it's easy enough to switch to
> using an org-font-lock-ensure variant that avoids these issues, so I
> will do that (in an updated patch and on Org's maint).

Thanks.

Kyle Meyer <kyle@kyleam.com> writes:

>> However, it's easy enough to switch to using an org-font-lock-ensure
>> variant that avoids these issues, so I will do that (in an updated
>> patch and on Org's maint).
>
> I've attached updated patches against emacs-25.  If these are applied to
> the Emacs repo, I'll backport the new font-lock-ensure alias to Org's
> maint.

Looks good, thanks.

Rasmus

-- 
Er du tosset for noge' lårt!

bug#22399: [O] bug#22399: org-mode fontification

[Phillip Lord]

Rasmus <rasmus@gmx.us> writes:
>
> Kyle Meyer <kyle@kyleam.com> writes:
>
>>> However, it's easy enough to switch to using an org-font-lock-ensure
>>> variant that avoids these issues, so I will do that (in an updated
>>> patch and on Org's maint).
>>
>> I've attached updated patches against emacs-25.  If these are applied to
>> the Emacs repo, I'll backport the new font-lock-ensure alias to Org's
>> maint.
>
> Looks good, thanks.


Just checking whether on who is going to apply the patches! They don't
seem to be on emacs-25 yet.

Phil

bug#22399: org-mode fontification

[Kyle Meyer]

[-- Attachment #1: Type: text/plain, Size: 902 bytes --]

phillip.lord@russet.org.uk (Phillip Lord) writes:

> Rasmus <rasmus@gmx.us> writes:
>>
>> Kyle Meyer <kyle@kyleam.com> writes:
>>
>>>> However, it's easy enough to switch to using an org-font-lock-ensure
>>>> variant that avoids these issues, so I will do that (in an updated
>>>> patch and on Org's maint).
>>>
>>> I've attached updated patches against emacs-25.  If these are applied to
>>> the Emacs repo, I'll backport the new font-lock-ensure alias to Org's
>>> maint.
>>
>> Looks good, thanks.
>
>
> Just checking whether on who is going to apply the patches! They don't
> seem to be on emacs-25 yet.

I have copyright assignment with the FSF (for Org mode contributions),
but I haven't contributed to Emacs directly before.  I'm cc'ing Stefan
because these patches are touching his changes.

I've re-attached the most recent set of patches
(http://permalink.gmane.org/gmane.emacs.bugs/111756).


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-compat-Fix-font-lock-ensure-alias.patch --]
[-- Type: text/x-patch, Size: 1049 bytes --]

From d9a8014e3919c328bf2b6acbe9d9127f3b06b5e1 Mon Sep 17 00:00:00 2001
From: Kyle Meyer <kyle@kyleam.com>
Date: Tue, 19 Jan 2016 12:53:04 -0500
Subject: [PATCH 1/2] org-compat: Fix font-lock-ensure alias

* lisp/org/org-compat.el (org-font-lock-ensure): Fix check for
font-lock-ensure.  Make compatibility function's arguments optional to
match with font-lock-ensure's.  (Bug#22399)
---
 lisp/org/org-compat.el | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lisp/org/org-compat.el b/lisp/org/org-compat.el
index 6403d4d..cf6aafc 100644
--- a/lisp/org/org-compat.el
+++ b/lisp/org/org-compat.el
@@ -479,9 +479,9 @@ org-no-popups
       (not (null pos)))))
 
 (defalias 'org-font-lock-ensure
-  (if (fboundp 'org-font-lock-ensure)
+  (if (fboundp 'font-lock-ensure)
       #'font-lock-ensure
-    (lambda (_beg _end) (font-lock-fontify-buffer))))
+    (lambda (&optional _beg _end) (font-lock-fontify-buffer))))
 
 (defun org-floor* (x &optional y)
   "Return a list of the floor of X and the fractional part of X.
-- 
2.7.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-org-src-Use-font-lock-fontify-buffer-for-blocks.patch --]
[-- Type: text/x-patch, Size: 1176 bytes --]

From 7b2e8cd23ee5326ca2b0845bed8a5d111a2a3b17 Mon Sep 17 00:00:00 2001
From: Kyle Meyer <kyle@kyleam.com>
Date: Tue, 19 Jan 2016 12:53:22 -0500
Subject: [PATCH 2/2] org-src: Use font-lock-fontify-buffer for blocks

* lisp/org/org-src.el (org-src-font-lock-fontify-block): Use
font-lock-fontify-buffer because font-lock-ensure does not display
source block fonts.

This corresponds to 003a0f10695f035e844d844eacb1a86a6d2df934 in the Org
repo.
---
 lisp/org/org-src.el | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lisp/org/org-src.el b/lisp/org/org-src.el
index 8e06aaa..5548f52 100644
--- a/lisp/org/org-src.el
+++ b/lisp/org/org-src.el
@@ -916,7 +916,9 @@ org-src-font-lock-fontify-block
 	    (delete-region (point-min) (point-max))
 	    (insert string " ") ;; so there's a final property change
 	    (unless (eq major-mode lang-mode) (funcall lang-mode))
-            (org-font-lock-ensure)
+	    ;; Avoid `font-lock-ensure', which does not display fonts in
+	    ;; source block.
+	    (font-lock-fontify-buffer)
 	    (setq pos (point-min))
 	    (while (setq next (next-single-property-change pos 'face))
 	      (put-text-property
-- 
2.7.0


[-- Attachment #4: Type: text/plain, Size: 10 bytes --]



--
Kyle

bug#22399: org-mode fontification

[Stefan Monnier]

>  (defalias 'org-font-lock-ensure
> -  (if (fboundp 'org-font-lock-ensure)
> +  (if (fboundp 'font-lock-ensure)
>        #'font-lock-ensure
> -    (lambda (_beg _end) (font-lock-fontify-buffer))))
> +    (lambda (&optional _beg _end) (font-lock-fontify-buffer))))

Looks good.

>  	    (unless (eq major-mode lang-mode) (funcall lang-mode))
> -            (org-font-lock-ensure)
> +	    ;; Avoid `font-lock-ensure', which does not display fonts in
> +	    ;; source block.
> +	    (font-lock-fontify-buffer)

Intuitively, this looks like a wrong fix.  Do you have a reproducible
recipe showing the problem that you're trying to fix?


        Stefan

bug#22399: org-mode fontification

[Kyle Meyer]

Thanks for having a look.

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>>  	    (unless (eq major-mode lang-mode) (funcall lang-mode))
>> -            (org-font-lock-ensure)
>> +	    ;; Avoid `font-lock-ensure', which does not display fonts in
>> +	    ;; source block.
>> +	    (font-lock-fontify-buffer)
>
> Intuitively, this looks like a wrong fix.  Do you have a reproducible
> recipe showing the problem that you're trying to fix?

Sure.  With

--8<---------------cut here---------------start------------->8---

#+BEGIN_SRC elisp
  (require 'cl-lib)
  (message "test")
#+END_SRC
--8<---------------cut here---------------end--------------->8---

in /tmp/scratch.org, run

  emacs -Q --eval "(setq org-src-fontify-natively t)" --visit /tmp/scratch.org

Go to the code block and hit C-c ' (org-edit-special), and then exit
with another C-c '.  Without this change, the source block is no longer
highlighted as elisp code when I return to the buffer.

I don't think it's just something odd on my end because, when we
backported 6711a21, a user reported that he lost highlighting in source
blocks.

  http://thread.gmane.org/gmane.emacs.orgmode/99495

--
Kyle

bug#22399: org-mode fontification

[Stefan Monnier]

> Sure.  With
>
> --8<---------------cut here---------------start------------->8---
>
> #+BEGIN_SRC elisp
>   (require 'cl-lib)
>   (message "test")
> #+END_SRC
> --8<---------------cut here---------------end--------------->8---
>
> in /tmp/scratch.org, run
>
>   emacs -Q --eval "(setq org-src-fontify-natively t)" --visit /tmp/scratch.org
>
> Go to the code block and hit C-c ' (org-edit-special), and then exit
> with another C-c '.  Without this change, the source block is no longer
> highlighted as elisp code when I return to the buffer.

I see the problem.  org-src-font-lock-fontify-block is using buffers
named " org-src-fontification:<major-mode>" in an unusual way:
they're updated via normal buffer modifications, but they're not put in
font-lock-mode, so all the font-lock machinery which tries to keep the
fontification up-to-date is short-circuited, so it triggers a bug in
font-lock-ensure where we made incorrect assumptions.

I installed the patch below into emacs-25, which should fix this
problem,


        Stefan


--- a/lisp/font-lock.el
+++ b/lisp/font-lock.el
@@ -1074,7 +1074,13 @@ font-lock-flush
 
 (defvar font-lock-ensure-function
   (lambda (_beg _end)
-    (unless font-lock-fontified (font-lock-default-fontify-buffer)))
+    (unless font-lock-fontified
+      (font-lock-default-fontify-buffer)
+      (unless font-lock-mode
+        ;; If font-lock is not enabled, we don't have the hooks in place to
+        ;; track modifications, so a subsequent call to font-lock-ensure can't
+        ;; assume that the fontification is still valid.
+        (setq font-lock-fontified nil))))
   "Function to make sure a region has been fontified.
 Called with two arguments BEG and END.")
 

bug#22399: org-mode fontification

[Kyle Meyer]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

[...]

> I see the problem.  org-src-font-lock-fontify-block is using buffers
> named " org-src-fontification:<major-mode>" in an unusual way:
> they're updated via normal buffer modifications, but they're not put in
> font-lock-mode, so all the font-lock machinery which tries to keep the
> fontification up-to-date is short-circuited, so it triggers a bug in
> font-lock-ensure where we made incorrect assumptions.
>
> I installed the patch below into emacs-25, which should fix this
> problem,

Thank you.  That seems to work nicely.

Is there anything else I need to do for the first patch?

-- 
Kyle

bug#22399: org-mode fontification

[Stefan Monnier]

>> I see the problem.  org-src-font-lock-fontify-block is using buffers
>> named " org-src-fontification:<major-mode>" in an unusual way:
>> they're updated via normal buffer modifications, but they're not put in
>> font-lock-mode, so all the font-lock machinery which tries to keep the
>> fontification up-to-date is short-circuited, so it triggers a bug in
>> font-lock-ensure where we made incorrect assumptions.
>> I installed the patch below into emacs-25, which should fix this
>> problem,
> Thank you.  That seems to work nicely.
> Is there anything else I need to do for the first patch?

No, it's good to go.  If you don't have commit rights, I can install it
for you into emacs-25, unless the Org guys prefer to do it via the
Org tree.


        Stefan

bug#22399: org-mode fontification

[Kyle Meyer]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Thank you.  That seems to work nicely.
>> Is there anything else I need to do for the first patch?
>
> No, it's good to go.  If you don't have commit rights, I can install it
> for you into emacs-25, unless the Org guys prefer to do it via the
> Org tree.

No, I don't have commit rights.  I'll update the Org side, but, for
issues that were introduced in the Emacs tree post 24.5, I don't know if
it's preferable to also update Emacs instead of waiting for the changes
to come over with a full sync.

(There's a similar situation in bug#22469.  Some issues were introduced
by changes to Org in the Emacs repo, and these were fixed in the Org
repo after the patch was backported.)

-- 
Kyle

bug#22399: org-mode fontification

[Stefan Monnier]

> No, I don't have commit rights.  I'll update the Org side, but, for
> issues that were introduced in the Emacs tree post 24.5, I don't know if
> it's preferable to also update Emacs instead of waiting for the changes
> to come over with a full sync.

Looks like there's no Org people following this, so I installed it into
emacs-25 (IOW it should be fixed in the next (25.0.91) pretest).


        Stefan

bug#22399: org-mode fontification

[Bastien Guerry]

Hi Stefan and Kyle,

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> No, I don't have commit rights.  I'll update the Org side, but, for
>> issues that were introduced in the Emacs tree post 24.5, I don't know if
>> it's preferable to also update Emacs instead of waiting for the changes
>> to come over with a full sync.
>
> Looks like there's no Org people following this, so I installed it into
> emacs-25 (IOW it should be fixed in the next (25.0.91) pretest).

Thanks for installing this -- Kyle, do you confirm you installed the
change in Org's git repo?

bug#22399: org-mode fontification

[Kyle Meyer]

Bastien Guerry <bastien.guerry@free.fr> writes:

> Hi Stefan and Kyle,
>
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>> No, I don't have commit rights.  I'll update the Org side, but, for
>>> issues that were introduced in the Emacs tree post 24.5, I don't know if
>>> it's preferable to also update Emacs instead of waiting for the changes
>>> to come over with a full sync.
>>
>> Looks like there's no Org people following this, so I installed it into
>> emacs-25 (IOW it should be fixed in the next (25.0.91) pretest).
>
> Thanks for installing this -- Kyle, do you confirm you installed the
> change in Org's git repo?

Yes, in a4edee4.

-- 
Kyle

bug#22399: org-mode fontification

[Bastien Guerry]

Kyle Meyer <kyle@kyleam.com> writes:

>> Thanks for installing this -- Kyle, do you confirm you installed the
>> change in Org's git repo?
>
> Yes, in a4edee4.

Thanks!

-- 
 Bastien

bug#22399: org-mode fontification

[Phillip Lord]

Bastien Guerry <bastien.guerry@free.fr> writes:

> Kyle Meyer <kyle@kyleam.com> writes:
>
>>> Thanks for installing this -- Kyle, do you confirm you installed the
>>> change in Org's git repo?
>>
>> Yes, in a4edee4.
>
> Thanks!


And thank you everyone for fixing!

Phil