[PATCH 0/3] synctex support for pdf export

[PATCH 0/3] synctex support for pdf export

[Aaron Ecay]

This patch series is an attempt to add synctex support to org mode.
With synctex, you can click on a spot in a document in a PDF viewer,
and have emacs jump to the line of code that generated that part of
the pdf.  (See http://emacswiki.org/emacs/AUCTeX for tips on getting
synctex set up with emacs).  This is very useful when authoring
documents -- you can notice a typo in the pdf, and click directly on
it to fix the corresponding line in the tex file.

This allows the same functionality with org files -- click on a spot
in the generated PDF, and jump in emacs to the org source for that
spot.  This functionality is currently controled by the variable
org-export-with-concordance (disabled by default).  It also relies on
the latex process generating a synctex file -- use \synctex=1 in the
document header to enable that (you probably have to use pdflatex,
xelatex, or lualatex -- not plain latex + dvipdf).

I have not tested this code extensively, but it does work for me.  I
don't know if it works for async export or not, since I haven't set up
a working environment for that.

There are currently limitations.  The granularity of the jumping is
not great, because of the way the parser works.  It will get you into
the paragraph corresponding to the PDF location, but no closer (with
pure latex, you will arrive at the exact line in the tex file).  You
also have to run org-latex-patch-synctex manually, unless you use the
direct-to-pdf export option (C-c C-e l p).  In regular latex, beamer
documents have somewhat degraded synctex granularity (in general, you
don't get to the exact source line, but only somewhere between
\begin{frame} and \end{frame}).  This may be compounded by the bad
granularity of this patch -- I have not tested this combo very much.

It should be possible to support the other direction of
synchronization as well -- jumping from a location in the org file to
a location in the PDF document (most pdf viewers show a highlight of
some sort, to pick out the location requested).  I haven't implemented
that yet, but it should be easy.  I have also not implemented support
for multi-file org documents (with #+include).  That is also in
principle possible, however it will be difficult (or at least tedious)
to implement in practice because it will require yet another round of
text properties to be applied at some point in the export process.

I'm releasing the code for comments on the implementation, which is
somewhat dense.  Because of the way the parser/exporter works, in
order to track the line number from beginning to end I need to use 2
kinds of text properties.  A change in the parser is also required, to
attach line numbers to objects.  (Though the implementation is dense,
it was also not terribly difficult to code, once I understood the code
I was modifying.  The clear structure of Nicolas's exporter/parser is
responsible for this fact, more than any cleverness of mine.)

Comments on the code are welcome.  Testers are also welcome.  In order
to test the code, you need to:

1) set up synctex to work with your emacs/pdf program
2) set org-export-with-concordance to t
3) make sure your latex program generates a .synctex.gz file
4) export your favorite org document to pdf (directly)
5) it should now work!

Thanks,
Aaron

Aaron Ecay (3):
  Add :begin-line and :end-line to parser results
  Introduce machinery to ox.el for concordance generation
  Add synctex modification based on concordance

 lisp/org-element.el | 120 +++++++++++++++++++++++++++++++++++++++++
 lisp/ox-latex.el    |  64 +++++++++++++++++++---
 lisp/ox.el          | 151 +++++++++++++++++++++++++++++++++++++++++-----------
 3 files changed, 297 insertions(+), 38 deletions(-)

-- 
1.8.2

[PATCH 1/3] Add :begin-line and :end-line to parser results

[Aaron Ecay]

* lisp/org-element.el: Add :begin-line and :end-line to parser results

These properties give the line on which an element begins/ends
---
 lisp/org-element.el | 120 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 120 insertions(+)

diff --git a/lisp/org-element.el b/lisp/org-element.el
index e09d2cb..c074092 100644
--- a/lisp/org-element.el
+++ b/lisp/org-element.el
@@ -521,6 +521,8 @@ Assume point is at the beginning of the block."
 		(nconc
 		 (list :begin begin
 		       :end end
+		       :begin-line (org-element--line-number-at-pos begin)
+		       :end-line (org-element--line-number-at-pos end)
 		       :hiddenp hidden
 		       :contents-begin contents-begin
 		       :contents-end contents-end
@@ -575,6 +577,8 @@ Assume point is at beginning of drawer."
 		(nconc
 		 (list :begin begin
 		       :end end
+		       :begin-line (org-element--line-number-at-pos begin)
+		       :end-line (org-element--line-number-at-pos end)
 		       :drawer-name name
 		       :hiddenp hidden
 		       :contents-begin contents-begin
@@ -635,6 +639,8 @@ Assume point is at beginning of dynamic block."
 		  (nconc
 		   (list :begin begin
 			 :end end
+			 :begin-line (org-element--line-number-at-pos begin)
+			 :end-line (org-element--line-number-at-pos end)
 			 :block-name name
 			 :arguments arguments
 			 :hiddenp hidden
@@ -696,6 +702,8 @@ Assume point is at the beginning of the footnote definition."
 	     (list :label label
 		   :begin begin
 		   :end end
+		   :begin-line (org-element--line-number-at-pos begin)
+		   :end-line (org-element--line-number-at-pos end)
 		   :contents-begin contents-begin
 		   :contents-end contents-end
 		   :post-blank (count-lines ending end)
@@ -812,6 +820,8 @@ Assume point is at beginning of the headline."
 		    (list :raw-value raw-value
 			  :begin begin
 			  :end end
+			  :begin-line (org-element--line-number-at-pos begin)
+			  :end-line (org-element--line-number-at-pos end)
 			  :pre-blank
 			  (if (not contents-begin) 0
 			    (count-lines pos-after-head contents-begin))
@@ -977,6 +987,8 @@ Assume point is at beginning of the inline task."
 		   (list :raw-value raw-value
 			 :begin begin
 			 :end end
+			 :begin-line (org-element--line-number-at-pos begin)
+			 :end-line (org-element--line-number-at-pos end)
 			 :hiddenp hidden
 			 :contents-begin contents-begin
 			 :contents-end contents-end
@@ -1096,6 +1108,8 @@ Assume point is at the beginning of the item."
 		  (list :bullet bullet
 			:begin begin
 			:end end
+			:begin-line (org-element--line-number-at-pos begin)
+			:end-line (org-element--line-number-at-pos end)
 			;; CONTENTS-BEGIN and CONTENTS-END may be
 			;; mixed up in the case of an empty item
 			;; separated from the next by a blank line.
@@ -1181,6 +1195,8 @@ Assume point is at the beginning of the list."
 	     (list :type type
 		   :begin begin
 		   :end end
+		   :begin-line (org-element--line-number-at-pos begin)
+		   :end-line (org-element--line-number-at-pos end)
 		   :contents-begin contents-begin
 		   :contents-end contents-end
 		   :structure struct
@@ -1238,6 +1254,8 @@ Assume point is at the beginning of the property drawer."
 		  (nconc
 		   (list :begin begin
 			 :end end
+			 :begin-line (org-element--line-number-at-pos begin)
+			 :end-line (org-element--line-number-at-pos end)
 			 :hiddenp hidden
 			 :contents-begin contents-begin
 			 :contents-end contents-end
@@ -1291,6 +1309,8 @@ Assume point is at the beginning of the block."
 		  (nconc
 		   (list :begin begin
 			 :end end
+			 :begin-line (org-element--line-number-at-pos begin)
+			 :end-line (org-element--line-number-at-pos end)
 			 :hiddenp hidden
 			 :contents-begin contents-begin
 			 :contents-end contents-end
@@ -1326,6 +1346,8 @@ and `:post-blank' keywords."
       (list 'section
 	    (list :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :contents-begin begin
 		  :contents-end pos-before-blank
 		  :post-blank (count-lines pos-before-blank end))))))
@@ -1381,9 +1403,14 @@ Assume point is at the beginning of the block."
 		   (list :type type
 			 :begin begin
 			 :end end
+			 :begin-line (org-element--line-number-at-pos begin)
+			 :end-line (org-element--line-number-at-pos end)
 			 :hiddenp hidden
 			 :contents-begin contents-begin
 			 :contents-end contents-end
+			 :contents (and contents-begin contents-end
+					(buffer-substring-no-properties
+					 contents-begin contents-end))
 			 :post-blank (count-lines pos-before-blank end)
 			 :post-affiliated post-affiliated)
 		   (cdr affiliated)))))))))
@@ -1437,6 +1464,8 @@ containing `:begin', `:end', `:info', `:post-blank' and
 	    (nconc
 	     (list :begin begin
 		   :end end
+		   :begin-line (org-element--line-number-at-pos begin)
+		   :end-line (org-element--line-number-at-pos end)
 		   :info info
 		   :post-blank (count-lines pos-before-blank end)
 		   :post-affiliated post-affiliated)
@@ -1488,6 +1517,8 @@ as keywords."
 		  :duration duration
 		  :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :post-blank post-blank)))))
 
 (defun org-element-clock-interpreter (clock contents)
@@ -1547,6 +1578,8 @@ Assume point is at comment beginning."
 	    (nconc
 	     (list :begin begin
 		   :end end
+		   :begin-line (org-element--line-number-at-pos begin)
+		   :end-line (org-element--line-number-at-pos end)
 		   :value value
 		   :post-blank (count-lines com-end end)
 		   :post-affiliated post-affiliated)
@@ -1596,6 +1629,8 @@ Assume point is at comment block beginning."
 		  (nconc
 		   (list :begin begin
 			 :end end
+			 :begin-line (org-element--line-number-at-pos begin)
+			 :end-line (org-element--line-number-at-pos end)
 			 :value value
 			 :hiddenp hidden
 			 :post-blank (count-lines pos-before-blank end)
@@ -1636,6 +1671,8 @@ containing `:begin', `:end', `:value', `:post-blank' and
 	     (list :value value
 		   :begin begin
 		   :end end
+		   :begin-line (org-element--line-number-at-pos begin)
+		   :end-line (org-element--line-number-at-pos end)
 		   :post-blank (count-lines pos-before-blank end)
 		   :post-affiliated post-affiliated)
 	     (cdr affiliated))))))
@@ -1708,6 +1745,8 @@ keywords."
 		  (nconc
 		   (list :begin begin
 			 :end end
+			 :begin-line (org-element--line-number-at-pos begin)
+			 :end-line (org-element--line-number-at-pos end)
 			 :value value
 			 :switches switches
 			 :number-lines number-lines
@@ -1772,6 +1811,8 @@ Assume point is at export-block beginning."
 		  (nconc
 		   (list :begin begin
 			 :end end
+			 :begin-line (org-element--line-number-at-pos begin)
+			 :end-line (org-element--line-number-at-pos end)
 			 :type type
 			 :value value
 			 :hiddenp hidden
@@ -1826,6 +1867,8 @@ Assume point is at the beginning of the fixed-width area."
 	    (nconc
 	     (list :begin begin
 		   :end end
+		   :begin-line (org-element--line-number-at-pos begin)
+		   :end-line (org-element--line-number-at-pos end)
 		   :value value
 		   :post-blank (count-lines end-area end)
 		   :post-affiliated post-affiliated)
@@ -1862,6 +1905,8 @@ keywords."
 	    (nconc
 	     (list :begin begin
 		   :end end
+		   :begin-line (org-element--line-number-at-pos begin)
+		   :end-line (org-element--line-number-at-pos end)
 		   :post-blank (count-lines post-hr end)
 		   :post-affiliated post-affiliated)
 	     (cdr affiliated))))))
@@ -1902,6 +1947,8 @@ containing `:key', `:value', `:begin', `:end', `:post-blank' and
 		   :value value
 		   :begin begin
 		   :end end
+		   :begin-line (org-element--line-number-at-pos begin)
+		   :end-line (org-element--line-number-at-pos end)
 		   :post-blank (count-lines pos-before-blank end)
 		   :post-affiliated post-affiliated)
 	     (cdr affiliated))))))
@@ -1948,6 +1995,8 @@ Assume point is at the beginning of the latex environment."
 		(nconc
 		 (list :begin begin
 		       :end end
+		       :begin-line (org-element--line-number-at-pos begin)
+		       :end-line (org-element--line-number-at-pos end)
 		       :value value
 		       :post-blank (count-lines code-end end)
 		       :post-affiliated code-begin)
@@ -1983,6 +2032,8 @@ keywords."
 		  :value value
 		  :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :post-blank (count-lines pos-before-blank end))))))
 
 (defun org-element-node-property-interpreter (node-property contents)
@@ -2080,6 +2131,8 @@ Assume point is at the beginning of the paragraph."
 	    (nconc
 	     (list :begin begin
 		   :end end
+		   :begin-line (org-element--line-number-at-pos begin)
+		   :end-line (org-element--line-number-at-pos end)
 		   :contents-begin contents-begin
 		   :contents-end contents-end
 		   :post-blank (count-lines before-blank end)
@@ -2127,6 +2180,8 @@ and `:post-blank' keywords."
 		  :scheduled scheduled
 		  :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :post-blank post-blank)))))
 
 (defun org-element-planning-interpreter (planning contents)
@@ -2172,6 +2227,8 @@ Assume point is at beginning of the section."
       (list 'quote-section
 	    (list :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :value value
 		  :post-blank (count-lines pos-before-blank end))))))
 
@@ -2260,6 +2317,8 @@ Assume point is at the beginning of the block."
 					  (org-trim parameters))
 			 :begin begin
 			 :end end
+			 :begin-line (org-element--line-number-at-pos begin)
+			 :end-line (org-element--line-number-at-pos end)
 			 :number-lines number-lines
 			 :preserve-indent preserve-indent
 			 :retain-labels retain-labels
@@ -2334,6 +2393,8 @@ Assume point is at the beginning of the table."
 	    (nconc
 	     (list :begin begin
 		   :end end
+		   :begin-line (org-element--line-number-at-pos begin)
+		   :end-line (org-element--line-number-at-pos end)
 		   :type type
 		   :tblfm tblfm
 		   ;; Only `org' tables have contents.  `table.el' tables
@@ -2389,6 +2450,8 @@ containing `:begin', `:end', `:contents-begin', `:contents-end',
 	    (list :type type
 		  :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :contents-begin contents-begin
 		  :contents-end contents-end
 		  :post-blank 0)))))
@@ -2436,6 +2499,8 @@ Assume point is at beginning of the block."
 		  (nconc
 		   (list :begin begin
 			 :end end
+			 :begin-line (org-element--line-number-at-pos begin)
+			 :end-line (org-element--line-number-at-pos end)
 			 :contents-begin contents-begin
 			 :contents-end contents-end
 			 :hiddenp hidden
@@ -2498,6 +2563,8 @@ Assume point is at the first star marker."
       (list 'bold
 	    (list :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :contents-begin contents-begin
 		  :contents-end contents-end
 		  :post-blank post-blank)))))
@@ -2551,6 +2618,8 @@ Assume point is at the first tilde marker."
 	    (list :value value
 		  :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :post-blank post-blank)))))
 
 (defun org-element-code-interpreter (code contents)
@@ -2589,6 +2658,8 @@ Assume point is at the beginning of the entity."
 		  :utf-8 (nth 6 value)
 		  :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :use-brackets-p bracketsp
 		  :post-blank post-blank)))))
 
@@ -2660,6 +2731,8 @@ Assume point is at the beginning of the snippet."
 		  :value value
 		  :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :post-blank post-blank)))))
 
 (defun org-element-export-snippet-interpreter (export-snippet contents)
@@ -2716,6 +2789,8 @@ and `:post-blank' as keywords."
 			:type type
 			:begin begin
 			:end end
+			:begin-line (org-element--line-number-at-pos begin)
+			:end-line (org-element--line-number-at-pos end)
 			:post-blank post-blank))))
       (org-element-put-property
        footnote-reference :inline-definition
@@ -2776,6 +2851,8 @@ Assume point is at the beginning of the babel call."
       (list 'inline-babel-call
 	    (list :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :info info
 		  :post-blank post-blank)))))
 
@@ -2837,6 +2914,8 @@ Assume point is at the beginning of the inline src block."
 		  :parameters parameters
 		  :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :post-blank post-blank)))))
 
 (defun org-element-inline-src-block-interpreter (inline-src-block contents)
@@ -2884,6 +2963,8 @@ Assume point is at the first slash marker."
       (list 'italic
 	    (list :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :contents-begin contents-begin
 		  :contents-end contents-end
 		  :post-blank post-blank)))))
@@ -2927,6 +3008,8 @@ Assume point is at the beginning of the latex fragment."
 	    (list :value value
 		  :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :post-blank post-blank)))))
 
 (defun org-element-latex-fragment-interpreter (latex-fragment contents)
@@ -3060,6 +3143,8 @@ Assume point is at the beginning of the link."
 		  :search-option search-option
 		  :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :contents-begin contents-begin
 		  :contents-end contents-end
 		  :post-blank post-blank)))))
@@ -3136,6 +3221,8 @@ Assume point is at the macro."
 		  :args args
 		  :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :post-blank post-blank)))))
 
 (defun org-element-macro-interpreter (macro contents)
@@ -3179,6 +3266,8 @@ Assume point is at the radio target."
       (list 'radio-target
 	    (list :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :contents-begin contents-begin
 		  :contents-end contents-end
 		  :post-blank post-blank
@@ -3221,6 +3310,8 @@ Assume point is at the beginning of the statistics-cookie."
       (list 'statistics-cookie
 	    (list :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :value value
 		  :post-blank post-blank)))))
 
@@ -3263,6 +3354,8 @@ Assume point is at the first plus sign marker."
       (list 'strike-through
 	    (list :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :contents-begin contents-begin
 		  :contents-end contents-end
 		  :post-blank post-blank)))))
@@ -3298,6 +3391,8 @@ Assume point is at the underscore."
       (list 'subscript
 	    (list :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :use-brackets-p bracketsp
 		  :contents-begin contents-begin
 		  :contents-end contents-end
@@ -3348,6 +3443,8 @@ Assume point is at the caret."
       (list 'superscript
 	    (list :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :use-brackets-p bracketsp
 		  :contents-begin contents-begin
 		  :contents-end contents-end
@@ -3377,6 +3474,8 @@ and `:post-blank' keywords."
     (list 'table-cell
 	  (list :begin begin
 		:end end
+		:begin-line (org-element--line-number-at-pos begin)
+		:end-line (org-element--line-number-at-pos end)
 		:contents-begin contents-begin
 		:contents-end contents-end
 		:post-blank 0))))
@@ -3415,6 +3514,8 @@ Assume point is at the target."
       (list 'target
 	    (list :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :value value
 		  :post-blank post-blank)))))
 
@@ -3518,6 +3619,8 @@ Assume point is at the beginning of the timestamp."
 			 :minute-end minute-end
 			 :begin begin
 			 :end end
+			 :begin-line (org-element--line-number-at-pos begin)
+			 :end-line (org-element--line-number-at-pos end)
 			 :post-blank post-blank)
 		   repeater-props)))))
 
@@ -3649,6 +3752,8 @@ Assume point is at the first underscore marker."
       (list 'underline
 	    (list :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :contents-begin contents-begin
 		  :contents-end contents-end
 		  :post-blank post-blank)))))
@@ -3680,6 +3785,8 @@ Assume point is at the first equal sign marker."
 	    (list :value value
 		  :begin begin
 		  :end end
+		  :begin-line (org-element--line-number-at-pos begin)
+		  :end-line (org-element--line-number-at-pos end)
 		  :post-blank post-blank)))))
 
 (defun org-element-verbatim-interpreter (verbatim contents)
@@ -4876,6 +4983,19 @@ end of ELEM-A."
 	      (cdr overlays)))
       (goto-char (org-element-property :end elem-B)))))
 
+(defun org-element--line-number-at-pos (pos)
+  "Return the buffer line number at POS, widening if necessary.
+
+In the case of a temporary buffer being prepared for export, this
+function looks for text properties (inserted if
+`org-export-with-concordance' is set to t) which allow it to
+return the line number in the original buffer, irrespective of
+changes the export process makes."
+  (or (get-text-property pos 'org-line-num-pre)
+      (save-excursion
+	(widen)
+	(line-number-at-pos pos))))
+
 (provide 'org-element)
 
 ;; Local variables:
-- 
1.8.2

[PATCH 2/3] Introduce machinery to ox.el for concordance generation

[Aaron Ecay]

* lisp/ox.el (org-export-concordance): New buffer-local variable
(org-export-with-concordance): New user option
(org-export--concordance-propertize),
(org-export--concordance-propertize-pre),
(org-export--build-concordance),
(org-export--read-concordance): New functions
(org-export-data): Call org-export–concordance-propertize where
appropriate
(org-export-as),
(org-export-to-buffer),
(org-export-to-file): Handle concordance generation

The general idea is as follows:
1) Before copying the buffer for export, call org-export--propertize-pre
   to add line-number properties to the buffer
2) The parser sees these properties during export (previous commit)
3) org-export-data adds text properties to the strings it generates,
   indicating which lines they originated from (using function
   org-export--propertize)
4) These properties survive into the output buffer (because export no
   longer calls org-no-properties)
5) After export is finished, org-export-build-concordance walks the
   result buffer, calculating a concordance of source lines and output
   lines.
6) This value is stored in the org buffer’s org-export-concordance local
   variable
---
 lisp/ox.el | 151 +++++++++++++++++++++++++++++++++++++++++++++++++------------
 1 file changed, 121 insertions(+), 30 deletions(-)

diff --git a/lisp/ox.el b/lisp/ox.el
index ff6407b..e1c76bd 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -302,6 +302,13 @@ and its CDR is a list of export options.")
 This marker will be used with `C-u C-c C-e' to make sure export repetition
 uses the same subtree if the previous command was restricted to a subtree.")
 
+(defvar org-export-concordance nil
+  "The concordance resulting from the last export operation.
+
+The variable is always buffer-local, and only manipulated if
+`org-export-with-concordance' is set.")
+(make-variable-buffer-local 'org-export-concordance)
+
 \f
 ;;; User-configurable Variables
 ;;
@@ -351,6 +358,16 @@ e.g. \"c:t\"."
   :group 'org-export-general
   :type 'boolean)
 
+(defcustom org-export-with-concordance nil
+  "Non-nil means to generate a concordance.
+
+This is currently useful only for the LaTeX backend (and derived
+backends), which can use it to patch the SyncTeX file generated
+by LaTeX, so that it is possible to jump back and forth between
+the org file and resulting pdf."
+  :group 'org-export-general
+  :type 'boolean)
+
 (defcustom org-export-with-creator 'comment
   "Non-nil means the postamble should contain a creator sentence.
 
@@ -2040,6 +2057,61 @@ INFO is a plist containing export directives."
       (let ((transcoder (cdr (assq type (plist-get info :translate-alist)))))
 	(and (functionp transcoder) transcoder)))))
 
+(defun org-export--concordance-propertize (data string)
+  "Add line number text properties to STRING, based on DATA.
+
+This will allow the construction of a concordance from the
+completed string."
+  (let ((len (length string)))
+    (when (> len 1)
+      (put-text-property 0 1 'org-line-num
+			 (org-element-property :begin-line data)
+			 string)
+      (put-text-property (1- len) len 'org-line-num
+			 (org-element-property :end-line data)
+			 string)))
+  string)
+
+(defun org-export--concordance-propertize-pre ()
+  "Put line-number text properties on a buffer.
+
+Each line gets a org-line-num-pre property, which is its line
+number in the buffer before any export operations have changed
+the buffer."
+  ;; This is called from `org-export-as', which has issued
+  ;; `save-restriction'.
+  (widen)
+  (while (= 0 (forward-line 1))
+    (put-text-property (point) (point-at-eol) 'org-line-num-pre
+		       (line-number-at-pos))))
+
+(defun org-export--build-concordance ()
+  "Build a concordance, based on text props in an exported buffer."
+  (save-excursion
+    (let ((res '())
+	  next)
+      (goto-char (point-min))
+      (while (setq next (next-single-property-change (point) 'org-line-num))
+	(goto-char next)
+	(setq res (cons (cons (line-number-at-pos)
+			      (get-text-property (point) 'org-line-num))
+			res))
+	(forward-char 1))
+      (setq res (nreverse res))
+      (setq next res)
+      (while (cdr next)
+	(if (equal (caar next) (caadr next))
+	    (setcdr next (cddr next))
+	  (setq next (cdr next))))
+      res)))
+
+(defun org-export--read-concordance (concordance src-line)
+  "Get the original line number from CONCORDANCE for output line SRC-LINE."
+  (while (and (caadr concordance)
+	      (<= (caadr concordance) src-line))
+    (setq concordance (cdr concordance)))
+  (cdar concordance))
+
 (defun org-export-data (data info)
   "Convert DATA into current back-end format.
 
@@ -2056,11 +2128,16 @@ Return transcoded string."
 	       ((memq data (plist-get info :ignore-list)) nil)
 	       ;; Plain text.
 	       ((eq type 'plain-text)
-		(org-export-filter-apply-functions
-		 (plist-get info :filter-plain-text)
-		 (let ((transcoder (org-export-transcoder data info)))
-		   (if transcoder (funcall transcoder data info) data))
-		 info))
+		(let* ((transcoder (org-export-transcoder data info))
+		       (transcoded-string (if transcoder
+					      (funcall transcoder data info)
+					    data))
+		       (propertized-string (org-export--concordance-propertize
+					    data transcoded-string)))
+		  (org-export-filter-apply-functions
+		   (plist-get info :filter-plain-text)
+		   propertized-string
+		   info)))
 	       ;; Uninterpreted element/object: change it back to Org
 	       ;; syntax and export again resulting raw string.
 	       ((not (org-export--interpret-p data info))
@@ -2081,15 +2158,18 @@ Return transcoded string."
 		    (and (eq type 'headline)
 			 (eq (plist-get info :with-archived-trees) 'headline)
 			 (org-element-property :archivedp data)))
-		(let ((transcoder (org-export-transcoder data info)))
-		  (or (and (functionp transcoder)
-			   (funcall transcoder data nil info))
-		      ;; Export snippets never return a nil value so
-		      ;; that white spaces following them are never
-		      ;; ignored.
-		      (and (eq type 'export-snippet) ""))))
-	       ;; Element/Object with contents.
-	       (t
+		(let* ((transcoder (org-export-transcoder data info))
+                       (transcoded-string
+                        (or (and (functionp transcoder)
+                                 (funcall transcoder data nil info))
+                            ;; Export snippets never return a nil value so
+                            ;; that white spaces following them are never
+                            ;; ignored.
+                            (and (eq type 'export-snippet) ""))))
+                  (and transcoded-string
+                       (org-export--concordance-propertize data transcoded-string))))
+               ;; Element/Object with contents.
+               (t
 		(let ((transcoder (org-export-transcoder data info)))
 		  (when transcoder
 		    (let* ((greaterp (memq type org-element-greater-elements))
@@ -2118,11 +2198,13 @@ Return transcoded string."
 					  data)
 				      (memq (org-element-type parent)
 					    '(footnote-definition item))))))))
-			     "")))
-		      (funcall transcoder data
-			       (if (not greaterp) contents
-				 (org-element-normalize-string contents))
-			       info))))))))
+			     ""))
+			   (transcoded-string
+			    (funcall transcoder data
+				     (if (not greaterp) contents
+				       (org-element-normalize-string contents))
+				     info)))
+		      (org-export--concordance-propertize data transcoded-string))))))))
 	;; Final result will be memoized before being returned.
 	(puthash
 	 data
@@ -2893,6 +2975,8 @@ Return code as a string."
 	;; created, where include keywords, macros are expanded and
 	;; code blocks are evaluated.
 	(org-export-with-buffer-copy
+	 (when org-export-with-concordance
+	   (org-export--concordance-propertize-pre))
 	 ;; Run first hook with current back-end as argument.
 	 (run-hook-with-args 'org-export-before-processing-hook backend)
 	 (org-export-expand-include-keyword)
@@ -2953,15 +3037,12 @@ Return code as a string."
 			     (funcall inner-template body info)))
 		(template (cdr (assq 'template
 				     (plist-get info :translate-alist)))))
-	   ;; Remove all text properties since they cannot be
-	   ;; retrieved from an external process.  Finally call
-	   ;; final-output filter and return result.
-	   (org-no-properties
-	    (org-export-filter-apply-functions
-	     (plist-get info :filter-final-output)
-	     (if (or (not (functionp template)) body-only) full-body
-	       (funcall template full-body info))
-	     info))))))))
+	   ;; Call final-output filter and return result.
+	   (org-export-filter-apply-functions
+	    (plist-get info :filter-final-output)
+	    (if (or (not (functionp template)) body-only) full-body
+	      (funcall template full-body info))
+	    info)))))))
 
 ;;;###autoload
 (defun org-export-to-buffer
@@ -2980,11 +3061,16 @@ see.
 Depending on `org-export-copy-to-kill-ring', add buffer contents
 to kill ring.  Return buffer."
   (let ((out (org-export-as backend subtreep visible-only body-only ext-plist))
-	(buffer (get-buffer-create buffer)))
+	(buffer (get-buffer-create buffer))
+	concordance)
     (with-current-buffer buffer
       (erase-buffer)
       (insert out)
+      (when org-export-with-concordance
+	(setq concordance (org-export--build-concordance)))
       (goto-char (point-min)))
+    (when concordance
+      (setq org-export-concordance concordance))
     ;; Maybe add buffer contents to kill ring.
     (when (and (org-export--copy-to-kill-ring-p) (org-string-nw-p out))
       (org-kill-new out))
@@ -3009,11 +3095,16 @@ to kill ring.  Return output file's name."
   ;; we'd rather avoid needless transcoding of parse tree.
   (unless (file-writable-p file) (error "Output file not writable"))
   ;; Insert contents to a temporary buffer and write it to FILE.
-  (let ((out (org-export-as backend subtreep visible-only body-only ext-plist)))
+  (let ((out (org-export-as backend subtreep visible-only body-only ext-plist))
+	concordance)
     (with-temp-buffer
       (insert out)
+      (when org-export-with-concordance
+	(setq concordance (org-export--build-concordance)))
       (let ((coding-system-for-write org-export-coding-system))
 	(write-file file)))
+    (when concordance
+      (setq org-export-concordance concordance))
     ;; Maybe add file contents to kill ring.
     (when (and (org-export--copy-to-kill-ring-p) (org-string-nw-p out))
       (org-kill-new out)))
-- 
1.8.2

[PATCH 3/3] Add synctex modification based on concordance

[Aaron Ecay]

* lisp/ox-latex.el (org-latex-patch-synctex): New function
(org-latex-export-to-pdf): Call it, if appropriate

This teaches the latex backend how to patch synctex files.  The patch
will be done automatically if exporting directly to pdf.  If not, the
user must call the org-latex-patch-synctex function (because it relies
on the synctex file, which will only be generated when pdflatex is run)

Touch the file after modifying the synctex, to trigger the pdf viewer to
re-read it.
---
 lisp/ox-latex.el | 64 +++++++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 56 insertions(+), 8 deletions(-)

diff --git a/lisp/ox-latex.el b/lisp/ox-latex.el
index 8727adc..43c5fde 100644
--- a/lisp/ox-latex.el
+++ b/lisp/ox-latex.el
@@ -115,6 +115,7 @@
 (eval-when-compile (require 'cl))
 (require 'ox)
 (require 'ox-publish)
+(require 'dired) 			; for dired-touch-program
 
 (defvar org-latex-default-packages-alist)
 (defvar org-latex-packages-alist)
@@ -1150,6 +1151,43 @@ just outside of it."
      (funcall search-refs element))
    ""))
 
+(defun org-latex-patch-synctex (&optional subtreep)
+  (cond
+   ((not org-export-concordance)
+	 (message "No concordance, not patching."))
+   ((not (file-exists-p
+	  (org-export-output-file-name ".synctex.gz" subtreep)))
+    (message "No synctex file found, not patching."))
+   (t
+    (let* ((conc org-export-concordance)
+	   (file-base (org-export-output-file-name "." subtreep))
+           (buf (find-file-noselect (concat file-base "synctex.gz"))))
+      (with-current-buffer buf
+	(let ((max-index 0)
+	      the-index extra-path)
+	  (goto-char (point-min))
+	  (while (re-search-forward "^Input:\\([0-9]+\\):" nil t)
+	    (setq max-index (max max-index (string-to-int (match-string 1)))))
+	  (goto-char (point-min))
+	  (when (re-search-forward (concat "^Input:\\([0-9]+\\):\\(.*\\)"
+					   (regexp-quote file-base) "tex$")
+				   nil t)
+	    (setq the-index (string-to-int (match-string 1)))
+	    (setq extra-path (match-string 2))
+	    (goto-char (line-end-position))
+	    (insert (format "\nInput:%s:%s%sorg" (1+ max-index) extra-path file-base)))
+	  (goto-char (point-min))
+	  (while (re-search-forward (format "^[vhxkgr$[)]\\(%s\\),\\([0-9]+\\):"
+					    the-index)
+				    nil t)
+            (replace-match (int-to-string (1+ max-index)) nil t nil 1)
+	    (replace-match
+	     (int-to-string
+	      (org-export--read-concordance conc (string-to-int (match-string 2))))
+	     nil t nil 2))
+	  (save-buffer)))
+      (kill-buffer buf)))))
+
 
 \f
 ;;; Template
@@ -2918,14 +2956,24 @@ Return PDF file's name."
       (let ((outfile (org-export-output-file-name ".tex" subtreep)))
 	(org-export-async-start
 	    (lambda (f) (org-export-add-to-stack f 'latex))
-	  `(expand-file-name
-	    (org-latex-compile
-	     (org-export-to-file
-	      'latex ,outfile ,subtreep ,visible-only ,body-only
-	      ',ext-plist)))))
-    (org-latex-compile
-     (org-latex-export-to-latex
-      nil subtreep visible-only body-only ext-plist))))
+	  `(let ((pdf-file
+		  (expand-file-name
+		   (org-latex-compile
+		    (org-export-to-file
+		     'latex ,outfile ,subtreep ,visible-only ,body-only
+		     ',ext-plist)))))
+	     (when org-export-with-concordance
+	       (org-latex-patch-synctex subtreep)
+	       (call-process dired-touch-program nil nil nil pdf-file))
+	     pdf-file)))
+    (let ((pdf-file
+	   (org-latex-compile
+	    (org-latex-export-to-latex
+	     nil subtreep visible-only body-only ext-plist))))
+      (when org-export-with-concordance
+	(org-latex-patch-synctex subtreep)
+        (call-process dired-touch-program nil nil nil pdf-file))
+      pdf-file)))
 
 (defun org-latex-compile (texfile &optional snippet)
   "Compile a TeX file.
-- 
1.8.2

Re: [PATCH 0/3] synctex support for pdf export

[Nicolas Goaziou]

Hello,

Aaron Ecay <aaronecay@gmail.com> writes:

> This patch series is an attempt to add synctex support to org mode.

Thank you for your patch.

> I have not tested this code extensively, but it does work for me.  I
> don't know if it works for async export or not, since I haven't set up
> a working environment for that.

Async export works out of-the-box (though not optimized). There's no
special environment to set up.

> There are currently limitations.  The granularity of the jumping is
> not great, because of the way the parser works.  It will get you into
> the paragraph corresponding to the PDF location, but no closer (with
> pure latex, you will arrive at the exact line in the tex file).  You
> also have to run org-latex-patch-synctex manually, unless you use the
> direct-to-pdf export option (C-c C-e l p).  In regular latex, beamer
> documents have somewhat degraded synctex granularity (in general, you
> don't get to the exact source line, but only somewhere between
> \begin{frame} and \end{frame}).  This may be compounded by the bad
> granularity of this patch -- I have not tested this combo very much.

[...]

As you notice, there are many limitations and I agree some of them will
be tedious to overcome. It also breaks asynchronous export.

Moreover, modifying both parser and core export framework for an
optional feature within a single back-end family is not right, IMO.

While I acknowledge the investment put into this patch, I won't accept
it in its current form. I might consider it if it only modifies
ox-latex.el, handles include keywords and buffer modifications through
Babel, and doesn't break asynchronous export. Not relying on text
properties is a real plus.

Though, don't push it too hard, I'm really not sure it's worth the
trouble.


Regards,

-- 
Nicolas Goaziou

Re: [PATCH 0/3] synctex support for pdf export

[Aaron Ecay]

Hi Nicolas,

2013ko apirilak 1an, Nicolas Goaziou-ek idatzi zuen:
> 
> Async export works out of-the-box (though not optimized). There's no
> special environment to set up.

For me, when I tried it the async emacs process died because it could
not find an external elisp library that I load from my init.el.  I
thought the problem was just a matter of me setting load-path
incorrectly or something, and never looked into it, since having async
export was not very important to me at the time (it just seemed like a
cool feature to try).

Now that this has come up, I have looked at it more.  It appears that
the /usr/share/emacs/site-lisp directory is not added to load-path in
the async export process.  I guess that it should be, since users’
init.el files could rely on libraries that are found there.

> 
> As you notice, there are many limitations and I agree some of them will
> be tedious to overcome. It also breaks asynchronous export.
> 
> Moreover, modifying both parser and core export framework for an
> optional feature within a single back-end family is not right, IMO.

I agree that this is suboptimal, yes.

> 
> While I acknowledge the investment put into this patch, I won't accept
> it in its current form. I might consider it if it only modifies
> ox-latex.el,

This will make the problem very difficult, if not impossible.  Generally
speaking, the buffer that the export functions see bears only a loose
relationship to the original buffer, since babel blocks, #+include
directives, etc. have changed the text.  I have tried to think of ways
to get around this fact, since working with the synctex file requires
knowing the original line number.  This is the best I could do.

My next idea is to use #+name properties on src blocks, tables, etc. to
try to line up the two buffers (:ID: properties could also be used, if
present).  However, this would be a pain, and I doubt it would work well
enough to justify itself.

Do you have any ideas about how this might be overcome?  What is needed
is to know, for any line in the exported output, which line of the org
file it corresponds to (within some small margin of error).

Thanks,

-- 
Aaron Ecay

Re: [PATCH 0/3] synctex support for pdf export

[Nicolas Goaziou]

Hello,

Aaron Ecay <aaronecay@gmail.com> writes:

> Now that this has come up, I have looked at it more.  It appears that
> the /usr/share/emacs/site-lisp directory is not added to load-path in
> the async export process.  I guess that it should be, since users’
> init.el files could rely on libraries that are found there.

This is debatable. I think that the sub-process shouldn't load too much
libraries. In your case, you effectively need to tweak
`org-export-async-init-file'.

> This will make the problem very difficult, if not impossible.  Generally
> speaking, the buffer that the export functions see bears only a loose
> relationship to the original buffer, since babel blocks, #+include
> directives, etc. have changed the text.  I have tried to think of ways
> to get around this fact, since working with the synctex file requires
> knowing the original line number.  This is the best I could do.

[...]

> Do you have any ideas about how this might be overcome?  What is needed
> is to know, for any line in the exported output, which line of the org
> file it corresponds to (within some small margin of error).

I don't think there is a complete solution to this problem.

Though `org-export-before-processing-hook' allows to work on an exact
copy of the buffer being exported before any file is included and any
Babel block executed. You can add text properties there. And with
`org-export-before-parsing-hook', you can check what parts of the buffer
are new (generated from include keywords, macros and blocks). From there
it may be possible to implement a lookup function which would find the
appropriate line in the original buffer.


Regards,

-- 
Nicolas Goaziou

Re: [PATCH 0/3] synctex support for pdf export

[Andreas Leha]

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> Hello,
>
> Aaron Ecay <aaronecay@gmail.com> writes:
>
>> This patch series is an attempt to add synctex support to org mode.
>
> Thank you for your patch.
>
>> I have not tested this code extensively, but it does work for me.  I
>> don't know if it works for async export or not, since I haven't set up
>> a working environment for that.
>
> Async export works out of-the-box (though not optimized). There's no
> special environment to set up.
>
>> There are currently limitations.  The granularity of the jumping is
>> not great, because of the way the parser works.  It will get you into
>> the paragraph corresponding to the PDF location, but no closer (with
>> pure latex, you will arrive at the exact line in the tex file).  You
>> also have to run org-latex-patch-synctex manually, unless you use the
>> direct-to-pdf export option (C-c C-e l p).  In regular latex, beamer
>> documents have somewhat degraded synctex granularity (in general, you
>> don't get to the exact source line, but only somewhere between
>> \begin{frame} and \end{frame}).  This may be compounded by the bad
>> granularity of this patch -- I have not tested this combo very much.
>
> [...]
>
> As you notice, there are many limitations and I agree some of them will
> be tedious to overcome. It also breaks asynchronous export.
>
> Moreover, modifying both parser and core export framework for an
> optional feature within a single back-end family is not right, IMO.
>
> While I acknowledge the investment put into this patch, I won't accept
> it in its current form. I might consider it if it only modifies
> ox-latex.el, handles include keywords and buffer modifications through
> Babel, and doesn't break asynchronous export. Not relying on text
> properties is a real plus.
>
> Though, don't push it too hard, I'm really not sure it's worth the
> trouble.
>
>
> Regards,


Hi Nicolas,

I understand all your points very well:  Too heavy changes for an
optional and still feature-incomplete patch targeting a sub-group of users.

But the provided functionality would be really handy for everybody
who writes larger documents (like a thesis) with org-mode and has to
incorporate comments made to the pdf.

This feature has been asked for on this list
https://lists.gnu.org/archive/html/emacs-orgmode/2010-08/msg01253.html
and on stackoverflow
http://stackoverflow.com/questions/9965049/how-to-use-synctex-with-org-mode

So I just want to contradict your last statement:  IMO it's worth the
trouble to get this working in orgmode.

Best,
Andreas

Re: [PATCH 0/3] synctex support for pdf export

[Alan Schmitt]

Hello,

Andreas Leha writes:

> I understand all your points very well:  Too heavy changes for an
> optional and still feature-incomplete patch targeting a sub-group of users.
>
> But the provided functionality would be really handy for everybody
> who writes larger documents (like a thesis) with org-mode and has to
> incorporate comments made to the pdf.
>
> This feature has been asked for on this list
> https://lists.gnu.org/archive/html/emacs-orgmode/2010-08/msg01253.html
> and on stackoverflow
> http://stackoverflow.com/questions/9965049/how-to-use-synctex-with-org-mode
>
> So I just want to contradict your last statement:  IMO it's worth the
> trouble to get this working in orgmode.

I would happily use such a features. I write many presentations in
org-mode, and I miss being able to simply click to go back to the right
place in the buffer.

Alan

Re: [PATCH 0/3] synctex support for pdf export

[Bastien]

Hi all,

even if the feature were asked by many many people, I don't think it
would be the right time to implement it.  The current parser needs to
settle down a bit, to be heavily debugged in its current form before
we can move on and modify it for another feature.

So maybe for Org 9.0 if someone tackles the challenge to amend Aaron's
series of patches.

2 cents,

-- 
 Bastien

Re: [PATCH 0/3] synctex support for pdf export

[Aaron Ecay]

Hi Nicolas,

2013ko apirilak 4an, Nicolas Goaziou-ek idatzi zuen:
> 
> I don't think there is a complete solution to this problem.
> 
> Though `org-export-before-processing-hook' allows to work on an exact
> copy of the buffer being exported before any file is included and any
> Babel block executed. You can add text properties there. And with
> `org-export-before-parsing-hook', you can check what parts of the buffer
> are new (generated from include keywords, macros and blocks). From there
> it may be possible to implement a lookup function which would find the
> appropriate line in the original buffer.

Thanks for your advice.  I haven’t had time to work more on these
patches, but I do think I see a way forward.  Hopefully I (or somebody
else!) will have some time soon to devote to the problem.

Thanks,

-- 
Aaron Ecay

Re: [PATCH 0/3] synctex support for pdf export

[Rasmus]

Aaron Ecay <aaronecay@gmail.com> writes:

> 2013ko apirilak 4an, Nicolas Goaziou-ek idatzi zuen:
>> 
>> I don't think there is a complete solution to this problem.
>> 
>> Though `org-export-before-processing-hook' allows to work on an exact
>> copy of the buffer being exported before any file is included and any
>> Babel block executed. You can add text properties there. And with
>> `org-export-before-parsing-hook', you can check what parts of the buffer
>> are new (generated from include keywords, macros and blocks). From there
>> it may be possible to implement a lookup function which would find the
>> appropriate line in the original buffer.
>
> Thanks for your advice.  I haven’t had time to work more on these
> patches, but I do think I see a way forward.  Hopefully I (or somebody
> else!) will have some time soon to devote to the problem.

I would be interested in a SyncTeX-based solution.  Thanks for your
work on this, Aaron.

–Rasmus

-- 
Vote for proprietary math!