Org babel does not work properly with included files

Org babel does not work properly with included files

[Robert Goldman]

[-- Attachment #1: Type: text/plain, Size: 7623 bytes --]


Here's the problem:  when org-babel goes to look for parameters when executing a source block, it looks for them in the parent source file, and not in the parent source file with the included materials.  Here is the function that goes awry:

(defmacro org-babel-exp-in-export-file (lang &rest body)
  (declare (indent 1))
  `(let* ((lang-headers (intern (concat "org-babel-default-header-args:" ,lang)))
	  (heading (nth 4 (ignore-errors (org-heading-components))))
	  (link (when org-current-export-file
		  (org-make-link-string
		   (if heading
		       (concat org-current-export-file "::" heading)
		     org-current-export-file))))
	  (export-buffer (current-buffer)) results)
     (when link
       ;; resolve parameters in the original file so that
       ;; headline and file-wide parameters are included, attempt
       ;; to go to the same heading in the original file
       (set-buffer (get-file-buffer org-current-export-file))
       (save-restriction
	 (condition-case nil
**	     (org-open-link-from-string link)
	   (error (when heading
		    (goto-char (point-min))
		    (re-search-forward (regexp-quote heading) nil t))))
	 (setq results ,@body))
       (set-buffer export-buffer)
       results)))

The line that blows up, AFAICT is the one marked with an asterisk.  It goes to
look in the file *without* incorporated materials, instead of the file *with*
incorporated materials.

I believe that what needs to happen is that the link must be searched for in the
export-buffer, rather than in org-current-export-file.

Unfortunately, I don't understand the code well enough to figure out how to make
this modification.  I believe that the first trick is to NOT concatenate
org-current-export-file onto the link string.  The second would be to change
org-open-link-from-string to something that looks for the link match in the
current buffer.

To see the bug, it should be sufficient to install the attached to .org files
into a temporary directory and then try to export foo.org to latex.  You may
need to tweak your configuration so that evaluation is enabled for sh scripts
(require 'ob-sh)

The error messages one gets are deeply cryptic.  To debug this I modified
org-link-search as follows.  I added a DEBUG call when the link is unfound.
This is, of course, not generally good practice, but you will want this debug
call in place when you test the export per my instructions above.  You will get
a debug window with a backtrace that should make things relatively clear.  Here
is the modified org-link-search --- you can just evaluate the definition before
testing the export.  To see what I've changed, just search for debug...

Cheers,
r

(defun org-link-search (s &optional type avoid-pos)
  "Search for a link search option.
If S is surrounded by forward slashes, it is interpreted as a
regular expression.  In org-mode files, this will create an `org-occur'
sparse tree.  In ordinary files, `occur' will be used to list matches.
If the current buffer is in `dired-mode', grep will be used to search
in all files.  If AVOID-POS is given, ignore matches near that position."
  (let ((case-fold-search t)
	(s0 (mapconcat 'identity (org-split-string s "[ \t\r\n]+") " "))
	(markers (concat "\\(?:" (mapconcat (lambda (x) (regexp-quote (car x)))
					    (append '(("") (" ") ("\t") ("\n"))
						    org-emphasis-alist)
					    "\\|") "\\)"))
	(pos (point))
	(pre nil) (post nil)
	words re0 re1 re2 re3 re4_ re4 re5 re2a re2a_ reall)
    (cond
     ;; First check if there are any special search functions
     ((run-hook-with-args-until-success 'org-execute-file-search-functions s))
     ;; Now try the builtin stuff
     ((and (equal (string-to-char s0) ?#)
	   (> (length s0) 1)
	   (save-excursion
	     (goto-char (point-min))
	     (and
	      (re-search-forward
	       (concat "^[ \t]*:CUSTOM_ID:[ \t]+" (regexp-quote (substring s0 1)) "[ \t]*$") nil t)
	      (setq type 'dedicated
		    pos (match-beginning 0))))
	   ;; There is an exact target for this
	   (goto-char pos)
	   (org-back-to-heading t)))
     ((save-excursion
	(goto-char (point-min))
	(and
	 (re-search-forward
	  (concat "<<" (regexp-quote s0) ">>") nil t)
	 (setq type 'dedicated
	       pos (match-beginning 0))))
      ;; There is an exact target for this
      (goto-char pos))
     ((and (string-match "^(\\(.*\\))$" s0)
	   (save-excursion
	     (goto-char (point-min))
	     (and
	      (re-search-forward
	       (concat "[^[]" (regexp-quote
			       (format org-coderef-label-format
				       (match-string 1 s0))))
	       nil t)
	      (setq type 'dedicated
		    pos (1+ (match-beginning 0))))))
      ;; There is a coderef target for this
      (goto-char pos))
     ((string-match "^/\\(.*\\)/$" s)
      ;; A regular expression
      (cond
       ((org-mode-p)
	(org-occur (match-string 1 s)))
       ;;((eq major-mode 'dired-mode)
       ;; (grep (concat "grep -n -e '" (match-string 1 s) "' *")))
       (t (org-do-occur (match-string 1 s)))))
     ((and (org-mode-p) org-link-search-must-match-exact-headline)
      (and (equal (string-to-char s) ?*) (setq s (substring s 1)))
      (goto-char (point-min))
      (cond
       ((let (case-fold-search)
	  (re-search-forward (format org-complex-heading-regexp-format
				     (regexp-quote s))
			     nil t))
	;; OK, found a match
	(setq type 'dedicated)
	(goto-char (match-beginning 0)))
       ((and (not org-link-search-inhibit-query)
	     (eq org-link-search-must-match-exact-headline 'query-to-create)
	     (debug nil (format "No match for target %s" s))
	     (y-or-n-p "No match - create this as a new heading? "))
	(goto-char (point-max))
	(or (bolp) (newline))
	(insert "* " s "\n")
	(beginning-of-line 0))
       (t
	(goto-char pos)
	(error "No match"))))
     (t
      ;; A normal search string
      (when (equal (string-to-char s) ?*)
	;; Anchor on headlines, post may include tags.
	(setq pre "^\\*+[ \t]+\\(?:\\sw+\\)?[ \t]*"
	      post (org-re "[ \t]*\\(?:[ \t]+:[[:alnum:]_@#%:+]:[ \t]*\\)?$")
	      s (substring s 1)))
      (remove-text-properties
       0 (length s)
       '(face nil mouse-face nil keymap nil fontified nil) s)
      ;; Make a series of regular expressions to find a match
      (setq words (org-split-string s "[ \n\r\t]+")

	    re0 (concat "\\(<<" (regexp-quote s0) ">>\\)")
	    re2 (concat markers "\\(" (mapconcat 'downcase words "[ \t]+")
			"\\)" markers)
	    re2a_ (concat "\\(" (mapconcat 'downcase words "[ \t\r\n]+") "\\)[ \t\r\n]")
	    re2a (concat "[ \t\r\n]" re2a_)
	    re4_ (concat "\\(" (mapconcat 'downcase words "[^a-zA-Z_\r\n]+") "\\)[^a-zA-Z_]")
	    re4 (concat "[^a-zA-Z_]" re4_)

	    re1 (concat pre re2 post)
	    re3 (concat pre (if pre re4_ re4) post)
	    re5 (concat pre ".*" re4)
	    re2 (concat pre re2)
	    re2a (concat pre (if pre re2a_ re2a))
	    re4 (concat pre (if pre re4_ re4))
	    reall (concat "\\(" re0 "\\)\\|\\(" re1 "\\)\\|\\(" re2
			  "\\)\\|\\(" re3 "\\)\\|\\(" re4 "\\)\\|\\("
			  re5 "\\)"
			  ))
      (cond
       ((eq type 'org-occur) (org-occur reall))
       ((eq type 'occur) (org-do-occur (downcase reall) 'cleanup))
       (t (goto-char (point-min))
	  (setq type 'fuzzy)
	  (if (or (and (org-search-not-self 1 re0 nil t) (setq type 'dedicated))
		  (org-search-not-self 1 re1 nil t)
		  (org-search-not-self 1 re2 nil t)
		  (org-search-not-self 1 re2a nil t)
		  (org-search-not-self 1 re3 nil t)
		  (org-search-not-self 1 re4 nil t)
		  (org-search-not-self 1 re5 nil t)
		  )
	      (goto-char (match-beginning 1))
	    (goto-char pos)
	    (error "No match"))))))
    (and (org-mode-p) (org-show-context 'link-search))
    type))

[-- Attachment #2: foo.org --]
[-- Type: text/plain, Size: 170 bytes --]

* Purpose

This document is intended to demonstrate that src buffer evaluation in
subsidiary, included files, does not work.

* Demo

#+include "~/src/org-test/bar.org" 

[-- Attachment #3: bar.org --]
[-- Type: text/plain, Size: 89 bytes --]

* Here's the demo

#+begin_src sh :exports results :results output
echo "foo!"
#+end_src

Re: Org babel does not work properly with included files

[Nick Dokos]

Robert Goldman <rpgoldman@sift.info> wrote:

foo.org:
> * Purpose
> 
> This document is intended to demonstrate that src buffer evaluation in
> subsidiary, included files, does not work.
> 
> * Demo
> 
> #+include "~/src/org-test/bar.org" 
> 
> 

bar.org:
> * Here's the demo
> 
> #+begin_src sh :exports results :results output
> echo "foo!"
> #+end_src
> 

As a workaround, try making the headings identical, i.e. bar.org looks
like this:

--8<---------------cut here---------------start------------->8---
* Demo

#+begin_src sh :exports results :results output
echo "foo!"
#+end_src


--8<---------------cut here---------------end--------------->8---

Nick

Re: Org babel does not work properly with included files

[Robert P. Goldman]

[-- Attachment #1: Type: text/plain, Size: 1150 bytes --]

Sorry, I think I have created a red herring here by leaving that code block in both files. To see what the problem really is, consider the case where the source code block appears ONLY in the included file.

(I tested the source block in the master file to make sure it worked before I copied it into the included file and forgot to remove it from the master file.)

Best,
R
-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Nick Dokos <nicholas.dokos@hp.com> wrote:

Robert Goldman <rpgoldman@sift.info> wrote: foo.org: > * Purpose > > This document is intended to demonstrate that src buffer evaluation in > subsidiary, included files, does not work. > > * Demo > > #+include "~/src/org-test/bar.org" > > bar.org: > * Here's the demo > > #+begin_src sh :exports results :results output > echo "foo!" > #+end_src > As a workaround, try making the headings identical, i.e. bar.org looks like this: --8<---------------cut here---------------start------------->8--- * Demo #+begin_src sh :exports results :results output echo "foo!" #+end_src --8<---------------cut here---------------end--------------->8--- Nick 


[-- Attachment #2: Type: text/html, Size: 1689 bytes --]

Re: Org babel does not work properly with included files

[Nick Dokos]

Robert P. Goldman <rpgoldman@sift.info> wrote:


> Sorry, I think I have created a red herring here by leaving that code blo=
> ck in both files. To see what the problem really is, consider the case wh=
> ere the source code block appears ONLY in the included file.
> 
> (I tested the source block in the master file to make sure it worked befo=
> re I copied it into the included file and forgot to remove it from the ma=
> ster file.)
> 

??

There was no code block in foo.org: just the include.

Only bar.org had the code block.

As for multiple code blocks in the included file (your next message),
yeah, probably. But my suggestion was just a (possible) workaround:
it's definitely not a fix.

Nick

Re: Org babel does not work properly with included files

[Robert P. Goldman]

[-- Attachment #1: Type: text/plain, Size: 1079 bytes --]

Yes. That's right, the single source block was a simplification for the example. It was abstracted from a more complicated case with a dozen or so source blocks in the subordinate filter. But thanks for the suggestion.

Hope we can find a fix!

Best
R
-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Nick Dokos <nicholas.dokos@hp.com> wrote:

Robert P. Goldman <rpgoldman@sift.info> wrote: > Sorry, I think I have created a red herring here by leaving that code blo= > ck in both files. To see what the problem really is, consider the case wh= > ere the source code block appears ONLY in the included file. > > (I tested the source block in the master file to make sure it worked befo= > re I copied it into the included file and forgot to remove it from the ma= > ster file.) > ?? There was no code block in foo.org: just the include. Only bar.org had the code block. As for multiple code blocks in the included file (your next message), yeah, probably. But my suggestion was just a (possible) workaround: it's definitely not a fix. Nick 


[-- Attachment #2: Type: text/html, Size: 1510 bytes --]

Re: Org babel does not work properly with included files

[Robert P. Goldman]

[-- Attachment #1: Type: text/plain, Size: 885 bytes --]

Also note that if you have an included file with MULTIPLE code blocks, this approach won't generalize....
Best,

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Nick Dokos <nicholas.dokos@hp.com> wrote:

Robert Goldman <rpgoldman@sift.info> wrote: foo.org: > * Purpose > > This document is intended to demonstrate that src buffer evaluation in > subsidiary, included files, does not work. > > * Demo > > #+include "~/src/org-test/bar.org" > > bar.org: > * Here's the demo > > #+begin_src sh :exports results :results output > echo "foo!" > #+end_src > As a workaround, try making the headings identical, i.e. bar.org looks like this: --8<---------------cut here---------------start------------->8--- * Demo #+begin_src sh :exports results :results output echo "foo!" #+end_src --8<---------------cut here---------------end--------------->8--- Nick 


[-- Attachment #2: Type: text/html, Size: 1416 bytes --]

Re: Org babel does not work properly with included files

[Robert Goldman]

Looking over this some more, I see that the challenge is to:

1.  read the file parameters (whatever they are) from the original file
(hence opening the file from the link) and

2.  read the header parameters from the export buffer, since the header
may not actually be contained in the original file.

This seems like a substantial reorganization from the original, which
attempts to do both tasks in the original file (and fails for included
files).

best,
r

Re: Org babel does not work properly with included files

[Eric Schulte]

Robert Goldman <rpgoldman@sift.info> writes:

> Looking over this some more, I see that the challenge is to:
>
> 1.  read the file parameters (whatever they are) from the original file
> (hence opening the file from the link) and
>
> 2.  read the header parameters from the export buffer, since the header
> may not actually be contained in the original file.
>

The above is a good summary.  Babel ensures that code blocks will be
evaluated in the original buffer, so that they can e.g., reference a
code block outside of the exported subtree when only exporting a
subtree.

>
> This seems like a substantial reorganization from the original, which
> attempts to do both tasks in the original file (and fails for included
> files).
>

I'm not sure that the current behavior is a bug.  Is it reasonable to
place code block parameters into an included file?  These parameters
would not be successfully found during interactive evaluation, and could
only plausibly be used during export as you anticipated.

Also, this would seem to break Babel's current guarantee that code
blocks will be evaluated in the original Org-mode file.

If this were to be implemented, I think one option would be to perform
the following on export
1. make a copy of the original buffer
2. call `org-export-handle-include-files' in this copy
3. resolve all parameters in this copied buffer

The above shouldn't be difficult to implement, and creating the copy
shouldn't add too much computational time to the export process.

My concern is that expansion of included files may be more likely to
cause confusing behavior (e.g., by change the evaluation environment
between interactive evaluation and export) than it would to be used
constructively.

Maybe I have missed your initial use case.

Thanks -- Eric

p.s. Sorry about the delayed reply I've had very little mailing
     list connectivity lately

>
> best,
> r
>
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: Org babel does not work properly with included files

[Robert Goldman]

On 5/5/11 May 5 -11:56 AM, Eric Schulte wrote:
> Robert Goldman <rpgoldman@sift.info> writes:
> 
>> Looking over this some more, I see that the challenge is to:
>>
>> 1.  read the file parameters (whatever they are) from the original file
>> (hence opening the file from the link) and
>>
>> 2.  read the header parameters from the export buffer, since the header
>> may not actually be contained in the original file.
>>
> 
> The above is a good summary.  Babel ensures that code blocks will be
> evaluated in the original buffer, so that they can e.g., reference a
> code block outside of the exported subtree when only exporting a
> subtree.
> 
>>
>> This seems like a substantial reorganization from the original, which
>> attempts to do both tasks in the original file (and fails for included
>> files).
>>
> 
> I'm not sure that the current behavior is a bug.  Is it reasonable to
> place code block parameters into an included file?  These parameters
> would not be successfully found during interactive evaluation, and could
> only plausibly be used during export as you anticipated.

Aren't the code block parameters supposed to appear /with/ the code
block?  So here's the use case:

I have a file chapter.org.  This contains a full draft of a chapter of
my manual.  I finish it and circulate it for comments, then get it ready
for inclusion.

Now I have manual.org and I want to include the main body of chapter.org
(typically there's some front matter I leave off).

When I put the #include in manual.org, the source code snippets in
chapter.org, which used to work, no longer do.

This doesn't seem like /such/ a crazy use case that it shouldn't work,
does it?

[btw, I am not entirely sure I know what "header parameters" are --- are
these the parameters that come from the #+begin_src line?  If so,
shouldn't they definitely be read from the #+begin_src line?  They can't
very well be read from manual.org, which doesn't contain the #+begin_src
line.]

Sorry if I wasn't clear in my original message.

Best,
r

Re: Org babel does not work properly with included files

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 1313 bytes --]

>> 
>> I'm not sure that the current behavior is a bug.  Is it reasonable to
>> place code block parameters into an included file?  These parameters
>> would not be successfully found during interactive evaluation, and could
>> only plausibly be used during export as you anticipated.
>
> Aren't the code block parameters supposed to appear /with/ the code
> block?  So here's the use case:
>
> I have a file chapter.org.  This contains a full draft of a chapter of
> my manual.  I finish it and circulate it for comments, then get it ready
> for inclusion.
>
> Now I have manual.org and I want to include the main body of chapter.org
> (typically there's some front matter I leave off).
>
> When I put the #include in manual.org, the source code snippets in
> chapter.org, which used to work, no longer do.
>
> This doesn't seem like /such/ a crazy use case that it shouldn't work,
> does it?
>

Oh, my apologies, apparently in scanning this email thread I
mis-understood your use case.

I've just tried to re-create the situation you've described above
(including a file which contains code blocks).  I was unable to
reproduce your problem locally (the results of exporting both to html
and tex are included).  Could you modify the attached example
sufficiently to demonstrate the problem you're experiencing?


[-- Attachment #2: example.tar.bz2 --]
[-- Type: application/octet-stream, Size: 2318 bytes --]

[-- Attachment #3: Type: text/plain, Size: 848 bytes --]


>
> [btw, I am not entirely sure I know what "header parameters" are ---

see http://orgmode.org/manual/Header-arguments.html

> 
> are these the parameters that come from the #+begin_src line?  If so,
> shouldn't they definitely be read from the #+begin_src line?  They
> can't very well be read from manual.org, which doesn't contain the
> #+begin_src line.]
>

These could appear with the code block, or as a property in an enclosing
heading.

Another example of a case where the original file is needed rather than
the exported file would be the case of exporting a subtree which
includes a block which references a variable defined elsewhere in the
file.

>
> Sorry if I wasn't clear in my original message.
>

No problem, I believe I misread this email thread.

Thanks -- Eric

>
> Best,
> r
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: Org babel does not work properly with included files

[Robert Goldman]

On 5/5/11 May 5 -4:27 PM, Eric Schulte wrote:
>>>
>>> I'm not sure that the current behavior is a bug.  Is it reasonable to
>>> place code block parameters into an included file?  These parameters
>>> would not be successfully found during interactive evaluation, and could
>>> only plausibly be used during export as you anticipated.
>>
>> Aren't the code block parameters supposed to appear /with/ the code
>> block?  So here's the use case:
>>
>> I have a file chapter.org.  This contains a full draft of a chapter of
>> my manual.  I finish it and circulate it for comments, then get it ready
>> for inclusion.
>>
>> Now I have manual.org and I want to include the main body of chapter.org
>> (typically there's some front matter I leave off).
>>
>> When I put the #include in manual.org, the source code snippets in
>> chapter.org, which used to work, no longer do.
>>
>> This doesn't seem like /such/ a crazy use case that it shouldn't work,
>> does it?
>>
> 
> Oh, my apologies, apparently in scanning this email thread I
> mis-understood your use case.
> 
> I've just tried to re-create the situation you've described above
> (including a file which contains code blocks).  I was unable to
> reproduce your problem locally (the results of exporting both to html
> and tex are included).  Could you modify the attached example
> sufficiently to demonstrate the problem you're experiencing?

I will work on it --- I have a somewhat complicated perl script that is
pulling stuff out for me and exporting results as source.  Getting it to
work in a small example is tricky!
> 
>>
>> [btw, I am not entirely sure I know what "header parameters" are ---
> 
> see http://orgmode.org/manual/Header-arguments.html

That's what I thought --- I was just confused because the comment in the
code flips from using the term "argument" to "parameter," and I
considered the possibility that there were two different mechanisms.

Thanks,
r

Re: Org babel does not work properly with included files

[Robert Goldman]

I would say we can close this bug report as INVALID.

AFAICT, the actual problem was happening simply because of the incorrect
treatment of missing links during export, which is now fixed.  When that
got fixed, my issue went away.

best,
r