bug#22472: 25.0.50; org-mode: latex fragments can't be un-rendered after a revert

bug#22472: 25.0.50; org-mode: latex fragments can't be un-rendered after a revert

[Dima Kogan]

Hi. I'm using a recent emacs from git, and a mostly-recent org-mode from
debian (Debian version 8.3.2-1). There's a bug:

1. emacs -Q

2. open an org-mode buffer that contains a latex fragment, for instance
==========
123

$\sqrt x$

123
=========

3. C-c C-x C-l

   This renders the latex fragment to show the equation graphically. It
   also tells the user (in the mini-buffer) that C-c C-c will remove the
   rendering, and go back to text

4. M-x revert-buffer

   At this point I would expect the render to revert to text, but it
   doesn't. This isn't necessarily a bug. However...

5. C-c C-c

   Here I ask emacs to un-render the fragment, but it does nothing. If
   #4 isn't a bug, then this is definitely a bug.

Thanks!

bug#22472: 25.0.50; org-mode: latex fragments can't be un-rendered after a revert

[Bastien Guerry]

Hi Dima,

Dima Kogan <dima@secretsauce.net> writes:

> 3. C-c C-x C-l
>
>    This renders the latex fragment to show the equation graphically. It
>    also tells the user (in the mini-buffer) that C-c C-c will remove the
>    rendering, and go back to text

Latest stable version from Org does not use C-c C-c to remove the
rendering.

> 4. M-x revert-buffer
>
>    At this point I would expect the render to revert to text, but it
>    doesn't. This isn't necessarily a bug. However...

(Why would reverting the buffer unrender the LaTeX fragment if the
text file hasn't change?)

> 5. C-c C-c
>
>    Here I ask emacs to un-render the fragment, but it does nothing. If
>    #4 isn't a bug, then this is definitely a bug.

AFAIK C-c C-c is not used to remove the rendering anymore.

Any chance you can test with the stable Org from GNU ELPA?

Thanks,

-- 
 Bastien

bug#22472: 25.0.50; org-mode: latex fragments can't be un-rendered after a revert

[Dima Kogan]

Hi. Thank you for replying. I just tried with the latest daily snapshot
from org-mode.org, and this bug is still valid, although the problem at
least becomes recoverable without killing and re-opening the org buffer.


Bastien Guerry <bzg@gnu.org> writes:

> Dima Kogan <dima@secretsauce.net> writes:
>
>> 3. C-c C-x C-l
>>
>>    This renders the latex fragment to show the equation graphically. It
>>    also tells the user (in the mini-buffer) that C-c C-c will remove the
>>    rendering, and go back to text
>
> Latest stable version from Org does not use C-c C-c to remove the
> rendering.

Indeed C-c C-c no longer does this. There're no instructions in the
mini-buffer about how to unrender anymore. C-c C-x C-l is supposed
unrender anything that's rendered, and this works.


>> 4. M-x revert-buffer
>>
>>    At this point I would expect the render to revert to text, but it
>>    doesn't. This isn't necessarily a bug. However...
>
> (Why would reverting the buffer unrender the LaTeX fragment if the
> text file hasn't change?)

The latest org-mode still keeps the renders on a revert. This is a bug
because when a revert happens, emacs does not assume that the file
hasn't changed. In fact, it COULD have changed, so keeping the (possibly
out-of-date) render up is just wrong. Contrived example:

1. tst.org contains $\sqrt x$
2. C-c C-x C-l to render it
3. in the shell (outside of emacs): perl -p -i -e s/sqrt/sin/ tst.org
4. M-x revert-buffer
   Observe that the render stays up, even though the latex the render
   represents is no longer there

>> 5. C-c C-c
>>
>>    Here I ask emacs to un-render the fragment, but it does nothing. If
>>    #4 isn't a bug, then this is definitely a bug.

Right. But I can now use C-c C-x C-l again. Continuing instructions from
above

5. C-c C-x C-l
   One would expect this to unrender the equation, but org thinks that
   it is not rendered, so it attempts to render it. You get a new render
   of \sin x

6. C-c C-x C-l
   This is now an unrender to get back to the new text



So a revert puts emacs into an inconsistent state, where the equation is
rendered, but the source may or may not be there anymore, and where org
doesn't think there's a render there at all.

bug#22472: 25.0.50; org-mode: latex fragments can't be un-rendered after a revert

[Dima Kogan]

Dima Kogan <dima@secretsauce.net> writes:

> So a revert puts emacs into an inconsistent state, where the equation is
> rendered, but the source may or may not be there anymore, and where org
> doesn't think there's a render there at all.

This looks like an issue in emacs, rather than org. I think the right
solution is for a revert to delete all overlays. I.e.:

  (add-hook 'before-revert-hook 'delete-all-overlays)

This generally works, but is unideal because:


1. We'd want this to apply to org buffers, not to all emacs buffers

2. This works only if revert-buffer-function is nil or
revert-buffer--default.


If we don't want to delete overlays on revert, then
org-latex-fragment-image-overlays must be properly set to reflect the
overlay state, AND emacs should make sure the overlay is still valid on
revert.

Any particular thoughts?

bug#22472: 25.0.50; org-mode: latex fragments can't be un-rendered after a revert

[Nicolas Goaziou]

[-- Attachment #1: Type: text/plain, Size: 1034 bytes --]

Hello,

Dima Kogan <dima@secretsauce.net> writes:

> Dima Kogan <dima@secretsauce.net> writes:
>
>> So a revert puts emacs into an inconsistent state, where the equation is
>> rendered, but the source may or may not be there anymore, and where org
>> doesn't think there's a render there at all.
>
> This looks like an issue in emacs, rather than org. I think the right
> solution is for a revert to delete all overlays. I.e.:
>
>   (add-hook 'before-revert-hook 'delete-all-overlays)
>
> This generally works, but is unideal because:
>
>
> 1. We'd want this to apply to org buffers, not to all emacs buffers
>
> 2. This works only if revert-buffer-function is nil or
> revert-buffer--default.
>
>
> If we don't want to delete overlays on revert, then
> org-latex-fragment-image-overlays must be properly set to reflect the
> overlay state, AND emacs should make sure the overlay is still valid on
> revert.
>
> Any particular thoughts?

Would the following patch (applied on maint) solve the problem?


Regards,

-- 
Nicolas Goaziou

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Remove-LaTeX-overlay-when-text-below-is-modified.patch --]
[-- Type: text/x-patch, Size: 2642 bytes --]

From eb1de6c2bcd02593a4a704f805faa9020f5fdddb Mon Sep 17 00:00:00 2001
From: Nicolas Goaziou <mail@nicolasgoaziou.fr>
Date: Thu, 4 Feb 2016 09:32:51 +0100
Subject: [PATCH] Remove LaTeX overlay when text below is modified

* lisp/org.el (org--format-latex-make-overlay): New function.
(org-format-latex): Use new function.

Reported-by: Dima Kogan <dima@secretsauce.net>
<http://permalink.gmane.org/gmane.emacs.orgmode/104800>
---
 lisp/org.el | 41 ++++++++++++++++++++++-------------------
 1 file changed, 22 insertions(+), 19 deletions(-)

diff --git a/lisp/org.el b/lisp/org.el
index fd08292..a48ae70 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -19079,6 +19079,27 @@ for all fragments in the buffer."
 	  (set-window-start nil window-start)
 	  (message (concat msg "done")))))))
 
+(defun org--format-latex-make-overlay (beg end image)
+  "Build an overlay between BEG and END using IMAGE file.
+Register new overlay in `org-latex-fragment-image-overlays'."
+  (let ((ov (make-overlay beg end)))
+    (overlay-put ov 'org-overlay-type 'org-latex-overlay)
+    (overlay-put ov 'evaporate t)
+    (overlay-put ov
+		 'modification-hooks
+		 (list (lambda (o flag beg end &optional l)
+			 (unless flag
+			   (org-remove-latex-fragment-image-overlays
+			    (overlay-start o) (overlay-end o))))))
+    (if (featurep 'xemacs)
+	(progn
+	  (overlay-put ov 'invisible t)
+	  (overlay-put ov 'end-glyph (make-glyph (vector 'png :file image))))
+      (overlay-put ov
+		   'display
+		   (list 'image :type 'png :file image :ascent 'center)))
+    (push ov org-latex-fragment-image-overlays)))
+
 (defun org-format-latex
     (prefix &optional dir overlays msg forbuffer processing-type)
   "Replace LaTeX fragments with links to an image, and produce images.
@@ -19170,25 +19191,7 @@ Some of the options can be changed using the variable
 			     (when (eq (overlay-get o 'org-overlay-type)
 				       'org-latex-overlay)
 			       (delete-overlay o)))
-			   (let ((ov (make-overlay beg end)))
-			     (overlay-put ov
-					  'org-overlay-type
-					  'org-latex-overlay)
-			     (overlay-put ov 'evaporate t)
-			     (if (featurep 'xemacs)
-				 (progn
-				   (overlay-put ov 'invisible t)
-				   (overlay-put
-				    ov 'end-glyph
-				    (make-glyph
-				     (vector 'png :file movefile))))
-			       (overlay-put
-				ov 'display
-				(list 'image
-				      :type 'png
-				      :file movefile
-				      :ascent 'center)))
-			     (push ov org-latex-fragment-image-overlays))
+			   (org--format-latex-make-overlay beg end movefile)
 			   (goto-char end))
 		       (delete-region beg end)
 		       (insert
-- 
2.6.3

bug#22472: 25.0.50; org-mode: latex fragments can't be un-rendered after a revert

[Dima Kogan]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Would the following patch (applied on maint) solve the problem?

Hi. I didn't observe anything acting differently with this patch. What
did you see it do? Was it supposed to unrender on revert, or to keep the
render, but update the state?

bug#22472: 25.0.50; org-mode: latex fragments can't be un-rendered after a revert

[Nicolas Goaziou]

Dima Kogan <dima@secretsauce.net> writes:

> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> Would the following patch (applied on maint) solve the problem?
>
> Hi. I didn't observe anything acting differently with this patch. What
> did you see it do? Was it supposed to unrender on revert, or to keep the
> render, but update the state?

As the commit message says, it is supposed to remove the overlay when
text below is modified (e.g. when using a replace-regexp). E.g.,

 1. Write \(1 + 2\)

 2. Toggle overlay with C-c C-x C-l

 3. M-% 2 RET 3 RET

 4. The overlay should have disappeared

I didn't test it with the `revert-buffer' command, but I would be
surprised that it doesn't run any modification hook.

Regards,

-- 
Nicolas

bug#22472: 25.0.50; org-mode: latex fragments can't be un-rendered after a revert

[Dima Kogan]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Dima Kogan <dima@secretsauce.net> writes:
>
>> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>>
>>> Would the following patch (applied on maint) solve the problem?
>>
>> Hi. I didn't observe anything acting differently with this patch. What
>> did you see it do? Was it supposed to unrender on revert, or to keep the
>> render, but update the state?
>
> As the commit message says, it is supposed to remove the overlay when
> text below is modified (e.g. when using a replace-regexp). E.g.,
>
>  1. Write \(1 + 2\)
>
>  2. Toggle overlay with C-c C-x C-l
>
>  3. M-% 2 RET 3 RET
>
>  4. The overlay should have disappeared
>
> I didn't test it with the `revert-buffer' command, but I would be
> surprised that it doesn't run any modification hook.

Ah. Thanks for explaining. The patch helps somewhat, but one can still
get into an inconsistent state:

  1. Write \(1 + 2\)
  2. Toggle overlay with C-c C-x C-l
  3. M-x revert-buffer

After the revert, the overlay remains, which is arguably OK, since the
text has not changed. However after the revert org doesn't realize that
the overlay is still up: org-latex-fragment-image-overlays is nil.

I think the overlays should all disappear on a revert.

Thanks!

bug#22472: 25.0.50; org-mode: latex fragments can't be un-rendered after a revert

[Nicolas Goaziou]

Hello,

Dima Kogan <dima@secretsauce.net> writes:

> Ah. Thanks for explaining. The patch helps somewhat, but one can still
> get into an inconsistent state:
>
>   1. Write \(1 + 2\)
>   2. Toggle overlay with C-c C-x C-l
>   3. M-x revert-buffer
>
> After the revert, the overlay remains, which is arguably OK, since the
> text has not changed. However after the revert org doesn't realize that
> the overlay is still up: org-latex-fragment-image-overlays is nil.

Indeed. 

`org-latex-fragment-image-overlays' is a local variable and
`revert-buffer' calls `normal-mode', which, in turn, calls
`fundamental-mode'. The latter calls `kill-all-local-variables'.
Information is lost.

I don't think that `org-latex-fragment-image-overlays' is useful,
anyway. So I removed it, along with that bug, hopefully.

> I think the overlays should all disappear on a revert.

I'm not convinced that an unmodified overlay should disappear.

Thank you for the report.


Regards,

-- 
Nicolas Goaziou

bug#22472: bug#22472: 25.0.50; org-mode: latex fragments can't be un-rendered after a revert

[Stephen J. Barr]

[-- Attachment #1: Type: text/plain, Size: 1663 bytes --]

I was in the progress of writing my own email about this same issue. I was
not aware that C-c C-c is no longer used for disabling overlays. If this is
the case, what is the correct workflow for toggling these previews on and
off?

Thanks,

-Stephen



Stephen J. Barr
PhD Candidate, Operations Management
Dept: Information Systems and Operations Management (ISOM)
Michael G. Foster School of Business
University of Washington
Phone:  425 516 5012
Email: stevejb@uw.edu
Twitter: @stevejb
Website: http://stephen.planetbarr.com



On Fri, Feb 5, 2016 at 2:10 PM, Nicolas Goaziou <mail@nicolasgoaziou.fr>
wrote:

> Hello,
>
> Dima Kogan <dima@secretsauce.net> writes:
>
> > Ah. Thanks for explaining. The patch helps somewhat, but one can still
> > get into an inconsistent state:
> >
> >   1. Write \(1 + 2\)
> >   2. Toggle overlay with C-c C-x C-l
> >   3. M-x revert-buffer
> >
> > After the revert, the overlay remains, which is arguably OK, since the
> > text has not changed. However after the revert org doesn't realize that
> > the overlay is still up: org-latex-fragment-image-overlays is nil.
>
> Indeed.
>
> `org-latex-fragment-image-overlays' is a local variable and
> `revert-buffer' calls `normal-mode', which, in turn, calls
> `fundamental-mode'. The latter calls `kill-all-local-variables'.
> Information is lost.
>
> I don't think that `org-latex-fragment-image-overlays' is useful,
> anyway. So I removed it, along with that bug, hopefully.
>
> > I think the overlays should all disappear on a revert.
>
> I'm not convinced that an unmodified overlay should disappear.
>
> Thank you for the report.
>
>
> Regards,
>
> --
> Nicolas Goaziou
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 3161 bytes --]

bug#22472: 25.0.50; org-mode: latex fragments can't be un-rendered after a revert

[Dima Kogan]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Dima Kogan <dima@secretsauce.net> writes:
>
>> Ah. Thanks for explaining. The patch helps somewhat, but one can still
>> get into an inconsistent state:
>>
>>   1. Write \(1 + 2\)
>>   2. Toggle overlay with C-c C-x C-l
>>   3. M-x revert-buffer
>>
>> After the revert, the overlay remains, which is arguably OK, since the
>> text has not changed. However after the revert org doesn't realize that
>> the overlay is still up: org-latex-fragment-image-overlays is nil.
>
> Indeed. 
>
> `org-latex-fragment-image-overlays' is a local variable and
> `revert-buffer' calls `normal-mode', which, in turn, calls
> `fundamental-mode'. The latter calls `kill-all-local-variables'.
> Information is lost.
>
> I don't think that `org-latex-fragment-image-overlays' is useful,
> anyway. So I removed it, along with that bug, hopefully.

Yes, things look consistent after your patch.


>> I think the overlays should all disappear on a revert.
>
> I'm not convinced that an unmodified overlay should disappear.

Sure. If we can keep everything consistent, then keeping the overlays is
probably better.

Thank you very much! I'm closing this bug.