org tables into R?

org tables into R?

[Michael Gauland]

I want to use an R block to manipulate data from an org table, but I'm not
getting what I expect. 

If I define a table like this:

  #+NAME: data
  |   A |   B |  C |
  |-----+-----+----|
  | 115 |  76 | 60 |
  | 124 |  78 | 55 |
  | 118 |  73 | 65 |
  | 114 |  75 | 61 |
  | 108 |  74 | 82 |

and pass it into R like this:

  #+BEGIN_SRC R :results output :exports both :session :var data.table=data
  names(data.table)
  head(data.table)
  #+END_SRC

I expect to get a table with three columns ("A", "B", and C"), but instead I
seem to get a single column named "A.B.C":

  #+RESULTS:
  : [1] "A.B.C"
  :       A.B.C
  : 1 115 76 60
  : 2 124 78 55
  : 3 118 73 65
  : 4 114 75 61
  : 5 108 74 82


What am I doing wrong?

Thanks,
Michael Gauland

Re: org tables into R?

[Andreas Leha]

Hi Michael,

Michael Gauland <mikelygee@gmail.com> writes:
> I want to use an R block to manipulate data from an org table, but I'm not
> getting what I expect. 
>
> If I define a table like this:
>
>   #+NAME: data
>   |   A |   B |  C |
>
>   |-----+-----+----|
>   | 115 |  76 | 60 |
>   | 124 |  78 | 55 |
>   | 118 |  73 | 65 |
>   | 114 |  75 | 61 |
>   | 108 |  74 | 82 |
>
> and pass it into R like this:
>
>   #+BEGIN_SRC R :results output :exports both :session :var data.table=data
>   names(data.table)
>   head(data.table)
>   #+END_SRC
>
> I expect to get a table with three columns ("A", "B", and C"), but instead I
> seem to get a single column named "A.B.C":
>
>   #+RESULTS:
>   : [1] "A.B.C"
>   :       A.B.C
>   : 1 115 76 60
>   : 2 124 78 55
>   : 3 118 73 65
>   : 4 114 75 61
>   : 5 108 74 82
>
>
> What am I doing wrong?

I believe, that this is a bug that has been reported already [1] and is
still open.

Regards,
Andreas

[1] http://news.gmane.org/find-root.php?message_id=olulhmmtei5.fsf%40med.uni%2dgoettingen.de

Re: org tables into R?

[Vikas Rawal]

[-- Attachment #1: Type: text/plain, Size: 504 bytes --]


> Michael Gauland <mikelygee@gmail.com <mailto:mikelygee@gmail.com>> writes:
>> I want to use an R block to manipulate data from an org table, but I'm not
>> getting what I expect. 
>> 
>> 


>> 
>> What am I doing wrong?
> 
> I believe, that this is a bug that has been reported already [1] and is
> still open.
> 

The code sent by Michael works fine for me (that it, the org table is read as a data.frame).

It may help to upgrade your org-mode. What version are you on?

Vikas


[-- Attachment #2: Type: text/html, Size: 1737 bytes --]

Re: org tables into R?

[Michael Gauland]

I modified the definition of ob-R-transfer-variable-table-with-header in
ob-R.el by changing 'sep' from \"\\t\" to \"\". Now


  #+BEGIN_SRC R :results output :exports both :session :var data.table=data
:colnames yes
  names(data.table)
  head(data.table)
  #+END_SRC

(note the addition of :colnames=yes) give me:

  #+RESULTS:
  : [1] "A" "B" "C"
  :     A  B  C
  : 1 115 76 60
  : 2 124 78 55
  : 3 118 73 65
  : 4 114 75 61
  : 5 108 74 82

as I expect.

What is the current procedure to submit a patch?

Thanks,
Michael

Re: org tables into R?

[Andreas Leha]

Hi Vikas,

Thanks for following up on this issue!

Vikas Rawal <vikaslists@agrarianresearch.org> writes:
>     Michael Gauland <mikelygee@gmail.com> writes:
>     
>     I want to use an R block to manipulate data from an org table, but
>         I'm not
>         getting what I expect. 
>         
>         
>         
>
>         
>         What am I doing wrong?
>         
>
>     I believe, that this is a bug that has been reported already [1]
>     and is
>     still open.
>     
>     
>
> The code sent by Michael works fine for me (that it, the org table is
> read as a data.frame).
>
> It may help to upgrade your org-mode. What version are you on?
>
> Vikas

It turns out I cannot reproduce it either with emacs -Q.

I investigated and found that the culprit is an export filter I have
defined that replaces tabs with spaces:

--8<---------------cut here---------------start------------->8---
(defun my-e-beamer-final-filter (contents backend info)
  (replace-regexp-in-string "\t" "        " contents))
(add-to-list 'org-export-filter-final-output-functions 'my-e-beamer-final-filter)
--8<---------------cut here---------------end--------------->8---

Now, I consider that a bug as well.  I am not exporting anything, here.
But I am merely evaluating a code block.  Export filters should not
interfere.  Plus, it used to work (unfortunately, I do not have the time
to bisect right now).

So, why are export filters involved?  And can that be disabled?

Regards,
Andreas



PS: Here is my environment:

#+begin_src emacs-lisp
  (concat
   (replace-regexp-in-string " on .*" "" (emacs-version))
   "\n"
   (replace-regexp-in-string " @.*" "" (org-version nil t)))
#+end_src

#+results:
: GNU Emacs 24.4.50.1 (x86_64-apple-darwin13.3.0, NS appkit-1265.21 Version 10.9.4 (Build 13E28))
:  of 2014-09-02
: Org-mode version 8.3beta (release_8.3beta-709-g3e64f6

Re: org tables into R?

[Aaron Ecay]

[-- Attachment #1: Type: text/plain, Size: 1303 bytes --]

Hi Andreas,

Thanks for following up on this problem.

2015ko urtarrilak 4an, Andreas Leha-ek idatzi zuen:
> I investigated and found that the culprit is an export filter I have
> defined that replaces tabs with spaces:
> 
> --8<---------------cut here---------------start------------->8---
> (defun my-e-beamer-final-filter (contents backend info)
>   (replace-regexp-in-string "\t" "        " contents))
> (add-to-list 'org-export-filter-final-output-functions 'my-e-beamer-final-filter)
> --8<---------------cut here---------------end--------------->8---
> 
> Now, I consider that a bug as well.  I am not exporting anything, here.
> But I am merely evaluating a code block.  Export filters should not
> interfere.  Plus, it used to work (unfortunately, I do not have the time
> to bisect right now).
> 
> So, why are export filters involved?  And can that be disabled?

Recently(-ish), Nicolas updated the org-table functions to use the
export framework.  One of the drawbacks of this approach (as you have
brought to our attention) is that export filters are invoked.  This
should probably be disabled.

Another drawback is the detection of unexpanded macros (which will be
triggered e.g. if a tabular babel result has macro-like {{{text}}} in
it).  A proof-of-concept patch for this is attached.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-export-allow-sloppy-usage-of-macros-for-internal-API.patch --]
[-- Type: text/x-diff, Size: 1576 bytes --]

From 50ce44be96f446272d1e465c01265632f0ed488f Mon Sep 17 00:00:00 2001
From: Aaron Ecay <aaronecay@gmail.com>
Date: Sun, 4 Jan 2015 18:14:26 -0500
Subject: [PATCH] [export] allow sloppy usage of macros for internal APIs

* lisp/ox.el (org-export-as): Support :sloppy-macros plist entry.
* lisp/org-table.el (orgtbl-to-generic): Use it.
---
 lisp/org-table.el | 2 +-
 lisp/ox.el        | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/lisp/org-table.el b/lisp/org-table.el
index 6b33eda..7a53d7a 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -4834,7 +4834,7 @@ This may be either a string or a function of two arguments:
 	 (table-cell . ,(org-table--to-generic-cell params))
 	 ;; Section.  Return contents to avoid garbage around table.
 	 (section . (lambda (s c i) c))))
-      'body-only (org-combine-plists params '(:with-tables t))))))
+      'body-only (org-combine-plists params '(:with-tables t :sloppy-macros t))))))
 
 (defun org-table--generic-apply (value name &optional with-cons &rest args)
   (cond ((null value) nil)
diff --git a/lisp/ox.el b/lisp/ox.el
index f47baef..0fcfc04 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -2885,7 +2885,8 @@ Return code as a string."
 		(cons "email" (or (plist-get info :email) ""))
 		(cons "title"
 		      (org-element-interpret-data (plist-get info :title))))
-	  'finalize)
+	  (unless (plist-get info :sloppy-macros)
+	    'finalize))
 	 ;; Parse buffer.
 	 (setq tree (org-element-parse-buffer nil visible-only))
 	 ;; Handle left-over uninterpreted elements or objects in
-- 
2.2.1


[-- Attachment #3: Type: text/plain, Size: 681 bytes --]


I wonder, though, whether the various exceptions like this that will
stack up will tip the balance towards its being simpler for org-table to
have its own set of functions that operate on the org-element parse
tree, but don’t invoke org-export code.  For csv this would be something
like (pseudocode):

(org-element-mapconcat
 (lambda (table-row)
   (org-element-mapconcat
    (lambda (table-cell) 
      (quote-for-csv (org-element-interpret table-cell))
    table-row
    ","))
 table
 "\n")

Nicolas, WDYT?  If you prefer to continue using the export framework, I
can add some commentary to the attached patch and install it.

Thanks,

-- 
Aaron Ecay

Re: org tables into R?

[Nicolas Goaziou]

Hello,

Aaron Ecay <aaronecay@gmail.com> writes:

> Recently(-ish), Nicolas updated the org-table functions to use the
> export framework.  One of the drawbacks of this approach (as you have
> brought to our attention) is that export filters are invoked.  This
> should probably be disabled.

Indeed. I removed user-defined filters and hooks. Thank you.

> Another drawback is the detection of unexpanded macros (which will be
> triggered e.g. if a tabular babel result has macro-like {{{text}}} in
> it).  A proof-of-concept patch for this is attached.

I don't think it needs to be fixed: macro would then be silently
dropped, which is a step backwards. OTOH "ob-R.el" should consider using
":raw t" parameter for its table conversion function.


Regards,

-- 
Nicolas Goaziou

Re: org tables into R?

[Aaron Ecay]

[-- Attachment #1: Type: text/plain, Size: 1285 bytes --]

Hi Nicolas,

2015ko urtarrilak 5an, Nicolas Goaziou-ek idatzi zuen:
> 
> Hello,
> 
> Aaron Ecay <aaronecay@gmail.com> writes:
> 
>> Recently(-ish), Nicolas updated the org-table functions to use the
>> export framework.  One of the drawbacks of this approach (as you have
>> brought to our attention) is that export filters are invoked.  This
>> should probably be disabled.
> 
> Indeed. I removed user-defined filters and hooks. Thank you.
> 
>> Another drawback is the detection of unexpanded macros (which will be
>> triggered e.g. if a tabular babel result has macro-like {{{text}}} in
>> it).  A proof-of-concept patch for this is attached.
> 
> I don't think it needs to be fixed: macro would then be silently
> dropped, which is a step backwards. 

You are correct about the silent dropping of macro-like text.  However,
with current master that case gives an undefined macro error, which is
even worse.  Try this (in emacs -Q with org and ESS in the load-path) to
see it:

#+name: foo
#+begin_src R
  c("foo","{{{bar}}}")
#+end_src

> OTOH "ob-R.el" should consider using ":raw t" parameter for its table
> conversion function.

I think :raw is needed in ‘org-babel-insert-result’ in addition to my
previous patch.  New patch attached.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-babel-fix-macros-in-tabular-output.patch --]
[-- Type: text/x-diff, Size: 2996 bytes --]

From 4d2985ba88d2ba0c35ff715a7285469e8040d4b0 Mon Sep 17 00:00:00 2001
From: Aaron Ecay <aaronecay@gmail.com>
Date: Sun, 4 Jan 2015 18:14:26 -0500
Subject: [PATCH] [babel] fix macros in tabular output

* lisp/ox.el (org-export-as): Support :sloppy-macros plist entry.
* lisp/org-table.el (orgtbl-to-generic): Use it.
* lisp/ob-R.el (org-babel-R-assign-elisp):
* lisp/ob-core.el (org-babel-insert-result): Use :raw argument to
org-table conversion.
---
 lisp/ob-R.el      | 2 +-
 lisp/ob-core.el   | 2 +-
 lisp/org-table.el | 2 +-
 lisp/ox.el        | 3 ++-
 4 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/lisp/ob-R.el b/lisp/ob-R.el
index 2470b4f..6f76aa5 100644
--- a/lisp/ob-R.el
+++ b/lisp/ob-R.el
@@ -239,7 +239,7 @@ This function is called by `org-babel-execute-src-block'."
 	     (min (if lengths (apply 'min lengths) 0)))
         ;; Ensure VALUE has an orgtbl structure (depth of at least 2).
         (unless (listp (car value)) (setq value (list value)))
-	(let ((file (orgtbl-to-tsv value '(:fmt org-babel-R-quote-tsv-field)))
+	(let ((file (orgtbl-to-tsv value '(:raw t :fmt org-babel-R-quote-tsv-field)))
 	      (header (if (or (eq (nth 1 value) 'hline) colnames-p)
 			  "TRUE" "FALSE"))
 	      (row-names (if rownames-p "1" "NULL")))
diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 93fcb2a..9ff83f2 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -2213,7 +2213,7 @@ code ---- the results are extracted in the syntax of the source
 					(lambda (el) (or (listp el) (eq el 'hline)))
 					result)
 				       result (list result))
-				   '(:fmt (lambda (cell) (format "%s" cell)))) "\n"))
+				   '(:raw t :fmt (lambda (cell) (format "%s" cell)))) "\n"))
 		  (goto-char beg) (when (org-at-table-p) (org-table-align)))
 		 ((and (listp result) (not (funcall proper-list-p result)))
 		  (insert (format "%s\n" result)))
diff --git a/lisp/org-table.el b/lisp/org-table.el
index 6b33eda..7a53d7a 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -4834,7 +4834,7 @@ This may be either a string or a function of two arguments:
 	 (table-cell . ,(org-table--to-generic-cell params))
 	 ;; Section.  Return contents to avoid garbage around table.
 	 (section . (lambda (s c i) c))))
-      'body-only (org-combine-plists params '(:with-tables t))))))
+      'body-only (org-combine-plists params '(:with-tables t :sloppy-macros t))))))
 
 (defun org-table--generic-apply (value name &optional with-cons &rest args)
   (cond ((null value) nil)
diff --git a/lisp/ox.el b/lisp/ox.el
index f47baef..0fcfc04 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -2885,7 +2885,8 @@ Return code as a string."
 		(cons "email" (or (plist-get info :email) ""))
 		(cons "title"
 		      (org-element-interpret-data (plist-get info :title))))
-	  'finalize)
+	  (unless (plist-get info :sloppy-macros)
+	    'finalize))
 	 ;; Parse buffer.
 	 (setq tree (org-element-parse-buffer nil visible-only))
 	 ;; Handle left-over uninterpreted elements or objects in
-- 
2.2.1


[-- Attachment #3: Type: text/plain, Size: 282 bytes --]


There are several usages of orgtbl-to-* functions in babel (R, awk,
gnuplot, shell, sqlite, sql).  Org-plot also uses one.  The attached
patch adds :raw to the ob-R call, but if I understand correctly probably
all of them should add it.  Should I do that?

Thanks,

-- 
Aaron Ecay

Re: org tables into R?

[Andreas Leha]

Hi Aaron and Nicolas,

Thanks for partly fixing this issue.  Unfortunately, when working with
source block interactively the issue persists for me.  If in the same
sample file I view the source block via 'C-c C-v v' and step through the
generated code line-by-line, the table is again not split into columns.

This time I immediately tried with 'emacs -Q' and I can reproduce the
issue.

Here is the ecm again:

--8<---------------cut here---------------start------------->8---
* test
#+NAME: data
|   A |  B |  C |
|-----+----+----|
| 115 | 76 | 60 |
| 124 | 78 | 55 |
| 118 | 73 | 65 |
| 114 | 75 | 61 |
| 108 | 74 | 82 |

and pass it into R like this, then evaluation (C-c C-c) works, but
stepping through the code generated for viewing (C-c C-v v) does not:

#+BEGIN_SRC R :results output :exports both :session :var data.table=data
  names(data.table)
  head(data.table)
#+END_SRC

#+results:
: [1] "A" "B" "C"
:     A  B  C
: 1 115 76 60
: 2 124 78 55
: 3 118 73 65
: 4 114 75 61
: 5 108 74 82

#+begin_src emacs-lisp
  (concat
   (replace-regexp-in-string " on .*" "" (emacs-version))
   "\n"
   (replace-regexp-in-string " @.*" ")" (org-version nil t))
   "\n"
   (replace-regexp-in-string "].*" "]" (ess-version)))
#+end_src

#+results:
: GNU Emacs 24.4.50.1 (x86_64-apple-darwin13.3.0, NS appkit-1265.21 Version 10.9.4 (Build 13E28))
:  of 2014-09-02
: Org-mode version 8.3beta (release_8.3beta-717-ga8096c)
: ess-version: 14.05 [git: 4283f1304a54502c42707b6a4ba347703f0992dd]
--8<---------------cut here---------------end--------------->8---


Here is what I see in the org babel preview:

--8<---------------cut here---------------start------------->8---
data.table <- local({
     con <- textConnection(
       "\"A\"   \"B\"   \"C\"
\"115\" \"76\"  \"60\"
\"124\" \"78\"  \"55\"
\"118\" \"73\"  \"65\"
\"114\" \"75\"  \"61\"
\"108\" \"74\"  \"82\""
     )
     res <- utils::read.table(
       con,
       header    = TRUE,
       row.names = NULL,
       sep       = "\t",
       as.is     = TRUE
     )
     close(con)
     res
   })
names(data.table)
head(data.table)
--8<---------------cut here---------------end--------------->8---


And finally, here is my R session, when I step through the code:

--8<---------------cut here---------------start------------->8---

R version 3.1.2 (2014-10-31) -- "Pumpkin Helmet"
Copyright (C) 2014 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin10.8.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> > options(STERM='iESS', str.dendrogram.last="'", editor='emacsclient', show.error.locations=TRUE)
> data.table <- local({
+      con <- textConnection(
+        "\"A\"   \"B\"   \"C\"
+ \"115\" \"76\"  \"60\"
+ \"124\" \"78\"  \"55\"
+ \"118\" \"73\"  \"65\"
+ \"114\" \"75\"  \"61\"
+ \"108\" \"74\"  \"82\""
+      )
+      res <- utils::read.table(
+        con,
+        header    = TRUE,
+        row.names = NULL,
+        sep       = "\t",
+        as.is     = TRUE
+      )
+      close(con)
+      res
+    })
> names(data.table)
[1] "A...B...C"
> head(data.table)
   A...B...C
1 115 76  60
2 124 78  55
3 118 73  65
4 114 75  61
5 108 74  82
--8<---------------cut here---------------end--------------->8---


Regards,
Andreas

Re: org tables into R?

[Rainer M Krug]

[-- Attachment #1: Type: text/plain, Size: 6045 bytes --]

Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:

> Hi Aaron and Nicolas,

Sorry for coming so late to this topic - I should try to figure out what
is causing the problem as I am the one responsible for this code...

>
> Thanks for partly fixing this issue.  Unfortunately, when working with
> source block interactively the issue persists for me.  If in the same
> sample file I view the source block via 'C-c C-v v' and step through the
> generated code line-by-line, the table is again not split into columns.
>
> This time I immediately tried with 'emacs -Q' and I can reproduce the
> issue.
>
> Here is the ecm again:
>
> * test
> #+NAME: data
> |   A |  B |  C |
> |-----+----+----|
> | 115 | 76 | 60 |
> | 124 | 78 | 55 |
> | 118 | 73 | 65 |
> | 114 | 75 | 61 |
> | 108 | 74 | 82 |
>
> and pass it into R like this, then evaluation (C-c C-c) works, but
> stepping through the code generated for viewing (C-c C-v v) does not:
>
> #+BEGIN_SRC R :results output :exports both :session :var data.table=data
>   names(data.table)
>   head(data.table)
> #+END_SRC
>
> #+results:
> : [1] "A" "B" "C"
> :     A  B  C
> : 1 115 76 60
> : 2 124 78 55
> : 3 118 73 65
> : 4 114 75 61
> : 5 108 74 82
>
> #+begin_src emacs-lisp
>   (concat
>    (replace-regexp-in-string " on .*" "" (emacs-version))
>    "\n"
>    (replace-regexp-in-string " @.*" ")" (org-version nil t))
>    "\n"
>    (replace-regexp-in-string "].*" "]" (ess-version)))
> #+end_src
>
> #+results:
> : GNU Emacs 24.4.50.1 (x86_64-apple-darwin13.3.0, NS appkit-1265.21 Version 10.9.4 (Build 13E28))
> :  of 2014-09-02
> : Org-mode version 8.3beta (release_8.3beta-717-ga8096c)
> : ess-version: 14.05 [git: 4283f1304a54502c42707b6a4ba347703f0992dd]
>

,----
| #+RESULTS:
| : GNU Emacs 24.4.1 (x86_64-apple-darwin14.0.0, Carbon Version 157 AppKit 1343.16)
| :  of 2014-11-24
| : Org-mode version 8.3beta (release_8.3beta-695-g390015)
| : ess-version: 14.09 [svn: 6041 (2014-09-13)]
`----


I get the same in the preview, but it works for me. I do not use -q.

,----
| > data.table <- local({
| +      con <- textConnection(
| +        "\"A\"	\"B\"	\"C\"
| + \"115\"	\"76\"	\"60\"
| + \"124\"	\"78\"	\"55\"
| + \"118\"	\"73\"	\"65\"
| + \"114\"	\"75\"	\"61\"
| + \"108\"	\"74\"	\"82\""
| +      )
| +      res <- utils::read.table(
| +        con,
| +        header    = TRUE,
| +        row.names = NULL,
| +        sep       = "\t",
| +        as.is     = TRUE
| +      )
| +      close(con)
| +      res
| +    })
| >   names(data.table)
| [1] "A" "B" "C"
| >   head(data.table)
|     A  B  C
| 1 115 76 60
| 2 124 78 55
| 3 118 73 65
| 4 114 75 61
| 5 108 74 82
| > 
`----


>
> Here is what I see in the org babel preview:
>

This might be due to temail, but you don't have tabs (\t) between your
entries below - could you verify if this is the case? If yes, this might
be a locale issue (would be strange)?

> data.table <- local({
>      con <- textConnection(
>        "\"A\"   \"B\"   \"C\"
> \"115\" \"76\"  \"60\"
> \"124\" \"78\"  \"55\"
> \"118\" \"73\"  \"65\"
> \"114\" \"75\"  \"61\"
> \"108\" \"74\"  \"82\""
>      )
>      res <- utils::read.table(
>        con,
>        header    = TRUE,
>        row.names = NULL,
>        sep       = "\t",
>        as.is     = TRUE
>      )
>      close(con)
>      res
>    })
> names(data.table)
> head(data.table)

One solution would be to insert (instead of the tab character) "\t" to
separate the fields?

You could try and modify the ob-R-transfer-variable-table-with-header -
From the help:

,----
| ob-R-transfer-variable-table-with-header is a variable defined in `ob-R.el'.
| Its value is
| "%s <- local({\n     con <- textConnection(\n       %S\n     )\n     res <- utils::read.table(\n       con,\n       header    = %s,\n       row.names = %s,\n       sep       = \"\\t\",\n       as.is     = TRUE\n     )\n     close(con)\n     res\n   })"
| 
|   This variable may be risky if used as a file-local variable.
| 
| Documentation:
| R code used to transfer a table defined as a variable from org to R.
| 
| This function is used when the table contains a header.
`----

This is passed to the (format) function and one might be able to change
something there.

Cheers,

Rainer

>
>
> And finally, here is my R session, when I step through the code:
>
>
> R version 3.1.2 (2014-10-31) -- "Pumpkin Helmet"
> Copyright (C) 2014 The R Foundation for Statistical Computing
> Platform: x86_64-apple-darwin10.8.0 (64-bit)
>
> R is free software and comes with ABSOLUTELY NO WARRANTY.
> You are welcome to redistribute it under certain conditions.
> Type 'license()' or 'licence()' for distribution details.
>
>   Natural language support but running in an English locale
>
> R is a collaborative project with many contributors.
> Type 'contributors()' for more information and
> 'citation()' on how to cite R or R packages in publications.
>
> Type 'demo()' for some demos, 'help()' for on-line help, or
> 'help.start()' for an HTML browser interface to help.
> Type 'q()' to quit R.
>
>> > options(STERM='iESS', str.dendrogram.last="'", editor='emacsclient', show.error.locations=TRUE)
>> data.table <- local({
> +      con <- textConnection(
> +        "\"A\"   \"B\"   \"C\"
> + \"115\" \"76\"  \"60\"
> + \"124\" \"78\"  \"55\"
> + \"118\" \"73\"  \"65\"
> + \"114\" \"75\"  \"61\"
> + \"108\" \"74\"  \"82\""
> +      )
> +      res <- utils::read.table(
> +        con,
> +        header    = TRUE,
> +        row.names = NULL,
> +        sep       = "\t",
> +        as.is     = TRUE
> +      )
> +      close(con)
> +      res
> +    })
>> names(data.table)
> [1] "A...B...C"
>> head(data.table)
>    A...B...C
> 1 115 76  60
> 2 124 78  55
> 3 118 73  65
> 4 114 75  61
> 5 108 74  82
>
>
> Regards,
> Andreas
>
>
>

-- 
Rainer M. Krug
email: Rainer<at>krugs<dot>de
PGP: 0x0F52F982

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 494 bytes --]

Re: org tables into R?

[Andreas Leha]

Hi Rainer,

Thanks for chiming in.

Rainer M Krug <Rainer@krugs.de> writes:
> Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:
>
>> Hi Aaron and Nicolas,
>
> Sorry for coming so late to this topic - I should try to figure out what
> is causing the problem as I am the one responsible for this code...
>
>>
>> Thanks for partly fixing this issue.  Unfortunately, when working with
>> source block interactively the issue persists for me.  If in the same
>> sample file I view the source block via 'C-c C-v v' and step through the
>> generated code line-by-line, the table is again not split into columns.
>>
>> This time I immediately tried with 'emacs -Q' and I can reproduce the
>> issue.
>>
>> Here is the ecm again:
>>
>> * test
>> #+NAME: data
>> |   A |  B |  C |
>> |-----+----+----|
>> | 115 | 76 | 60 |
>> | 124 | 78 | 55 |
>> | 118 | 73 | 65 |
>> | 114 | 75 | 61 |
>> | 108 | 74 | 82 |
>>
>> and pass it into R like this, then evaluation (C-c C-c) works, but
>> stepping through the code generated for viewing (C-c C-v v) does not:
>>
>> #+BEGIN_SRC R :results output :exports both :session :var data.table=data
>>   names(data.table)
>>   head(data.table)
>> #+END_SRC
>>
>> #+results:
>> : [1] "A" "B" "C"
>> :     A  B  C
>> : 1 115 76 60
>> : 2 124 78 55
>> : 3 118 73 65
>> : 4 114 75 61
>> : 5 108 74 82
>>
>> #+begin_src emacs-lisp
>>   (concat
>>    (replace-regexp-in-string " on .*" "" (emacs-version))
>>    "\n"
>>    (replace-regexp-in-string " @.*" ")" (org-version nil t))
>>    "\n"
>>    (replace-regexp-in-string "].*" "]" (ess-version)))
>> #+end_src
>>
>> #+results:
>> : GNU Emacs 24.4.50.1 (x86_64-apple-darwin13.3.0, NS appkit-1265.21 Version 10.9.4 (Build 13E28))
>> :  of 2014-09-02
>> : Org-mode version 8.3beta (release_8.3beta-717-ga8096c)
>> : ess-version: 14.05 [git: 4283f1304a54502c42707b6a4ba347703f0992dd]
>>
>
> ,----
> | #+RESULTS:
> | : GNU Emacs 24.4.1 (x86_64-apple-darwin14.0.0, Carbon Version 157 AppKit 1343.16)
> | :  of 2014-11-24
> | : Org-mode version 8.3beta (release_8.3beta-695-g390015)
> | : ess-version: 14.09 [svn: 6041 (2014-09-13)]
> `----
>
>
> I get the same in the preview, but it works for me. I do not use -q.
>
> ,----
> | > data.table <- local({
> | +      con <- textConnection(
> | +        "\"A\"	\"B\"	\"C\"
> | + \"115\"	\"76\"	\"60\"
> | + \"124\"	\"78\"	\"55\"
> | + \"118\"	\"73\"	\"65\"
> | + \"114\"	\"75\"	\"61\"
> | + \"108\"	\"74\"	\"82\""
> | +      )
> | +      res <- utils::read.table(
> | +        con,
> | +        header    = TRUE,
> | +        row.names = NULL,
> | +        sep       = "\t",
> | +        as.is     = TRUE
> | +      )
> | +      close(con)
> | +      res
> | +    })
> | >   names(data.table)
> | [1] "A" "B" "C"
> | >   head(data.table)
> |     A  B  C
> | 1 115 76 60
> | 2 124 78 55
> | 3 118 73 65
> | 4 114 75 61
> | 5 108 74 82
> | > 
> `----
>
>
>>
>> Here is what I see in the org babel preview:
>>
>
> This might be due to temail, but you don't have tabs (\t) between your
> entries below - could you verify if this is the case? If yes, this might
> be a locale issue (would be strange)?

Yes, there seem to be no tabs.  If there were tabs, R should parse the
table correctly, I guess.

(BTW: My custom filter replacing tabs with spaces is still disabled,
even though I pulled from master, which includes the recent

commit 70f965535b07c03a40792237de3a970d1cb57551
Author: Nicolas Goaziou <mail@nicolasgoaziou.fr>
Date:   Mon Jan 5 12:39:26 2015 +0100

    org-table: Ignore filters and hooks in radio tables
)


>
>> data.table <- local({
>>      con <- textConnection(
>>        "\"A\"   \"B\"   \"C\"
>> \"115\" \"76\"  \"60\"
>> \"124\" \"78\"  \"55\"
>> \"118\" \"73\"  \"65\"
>> \"114\" \"75\"  \"61\"
>> \"108\" \"74\"  \"82\""
>>      )
>>      res <- utils::read.table(
>>        con,
>>        header    = TRUE,
>>        row.names = NULL,
>>        sep       = "\t",
>>        as.is     = TRUE
>>      )
>>      close(con)
>>      res
>>    })
>> names(data.table)
>> head(data.table)
>
> One solution would be to insert (instead of the tab character) "\t" to
> separate the fields?
>
> You could try and modify the ob-R-transfer-variable-table-with-header -
> From the help:
>
> ,----
> | ob-R-transfer-variable-table-with-header is a variable defined in `ob-R.el'.
> | Its value is
> | "%s <- local({\n con <- textConnection(\n %S\n )\n res <-
> | utils::read.table(\n con,\n header = %s,\n row.names = %s,\n sep =
> | \"\\t\",\n as.is = TRUE\n )\n close(con)\n res\n })"
> | 
> |   This variable may be risky if used as a file-local variable.
> | 
> | Documentation:
> | R code used to transfer a table defined as a variable from org to R.
> | 
> | This function is used when the table contains a header.
> `----
>
> This is passed to the (format) function and one might be able to change
> something there.

Thanks for the hint.  Do I assume correctly, that this variable is used
when the code block is evaluated as well?  In that case, I am reluctant
to change it, as the evaluation now works.  The surprising thing is the
different behaviour for preview and evaluation.

I am extremely tight with my time right now, so that I won't be able to
do much experimentation/investigation right now, I am afraid.  But I
will try to still be helpful...

Thanks,
Andreas


>
> Cheers,
>
> Rainer
>
>>
>>
>> And finally, here is my R session, when I step through the code:
>>
>>
>> R version 3.1.2 (2014-10-31) -- "Pumpkin Helmet"
>> Copyright (C) 2014 The R Foundation for Statistical Computing
>> Platform: x86_64-apple-darwin10.8.0 (64-bit)
>>
>> R is free software and comes with ABSOLUTELY NO WARRANTY.
>> You are welcome to redistribute it under certain conditions.
>> Type 'license()' or 'licence()' for distribution details.
>>
>>   Natural language support but running in an English locale
>>
>> R is a collaborative project with many contributors.
>> Type 'contributors()' for more information and
>> 'citation()' on how to cite R or R packages in publications.
>>
>> Type 'demo()' for some demos, 'help()' for on-line help, or
>> 'help.start()' for an HTML browser interface to help.
>> Type 'q()' to quit R.
>>
>>> > options(STERM='iESS', str.dendrogram.last="'", editor='emacsclient', show.error.locations=TRUE)
>>> data.table <- local({
>> +      con <- textConnection(
>> +        "\"A\"   \"B\"   \"C\"
>> + \"115\" \"76\"  \"60\"
>> + \"124\" \"78\"  \"55\"
>> + \"118\" \"73\"  \"65\"
>> + \"114\" \"75\"  \"61\"
>> + \"108\" \"74\"  \"82\""
>> +      )
>> +      res <- utils::read.table(
>> +        con,
>> +        header    = TRUE,
>> +        row.names = NULL,
>> +        sep       = "\t",
>> +        as.is     = TRUE
>> +      )
>> +      close(con)
>> +      res
>> +    })
>>> names(data.table)
>> [1] "A...B...C"
>>> head(data.table)
>>    A...B...C
>> 1 115 76  60
>> 2 124 78  55
>> 3 118 73  65
>> 4 114 75  61
>> 5 108 74  82
>>
>>
>> Regards,
>> Andreas
>>
>>
>>

Re: org tables into R?

[Rainer M Krug]

[-- Attachment #1: Type: text/plain, Size: 9521 bytes --]

Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:

> Hi Rainer,
>
> Thanks for chiming in.

Hope I can help...

>
> Rainer M Krug <Rainer@krugs.de> writes:
>> Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:
>>
>>> Hi Aaron and Nicolas,
>>
>> Sorry for coming so late to this topic - I should try to figure out what
>> is causing the problem as I am the one responsible for this code...
>>
>>>
>>> Thanks for partly fixing this issue.  Unfortunately, when working with
>>> source block interactively the issue persists for me.  If in the same
>>> sample file I view the source block via 'C-c C-v v' and step through the
>>> generated code line-by-line, the table is again not split into columns.
>>>
>>> This time I immediately tried with 'emacs -Q' and I can reproduce the
>>> issue.
>>>
>>> Here is the ecm again:
>>>
>>> * test
>>> #+NAME: data
>>> |   A |  B |  C |
>>> |-----+----+----|
>>> | 115 | 76 | 60 |
>>> | 124 | 78 | 55 |
>>> | 118 | 73 | 65 |
>>> | 114 | 75 | 61 |
>>> | 108 | 74 | 82 |
>>>
>>> and pass it into R like this, then evaluation (C-c C-c) works, but
>>> stepping through the code generated for viewing (C-c C-v v) does not:
>>>
>>> #+BEGIN_SRC R :results output :exports both :session :var data.table=data
>>>   names(data.table)
>>>   head(data.table)
>>> #+END_SRC
>>>
>>> #+results:
>>> : [1] "A" "B" "C"
>>> :     A  B  C
>>> : 1 115 76 60
>>> : 2 124 78 55
>>> : 3 118 73 65
>>> : 4 114 75 61
>>> : 5 108 74 82
>>>
>>> #+begin_src emacs-lisp
>>>   (concat
>>>    (replace-regexp-in-string " on .*" "" (emacs-version))
>>>    "\n"
>>>    (replace-regexp-in-string " @.*" ")" (org-version nil t))
>>>    "\n"
>>>    (replace-regexp-in-string "].*" "]" (ess-version)))
>>> #+end_src
>>>
>>> #+results:
>>> : GNU Emacs 24.4.50.1 (x86_64-apple-darwin13.3.0, NS appkit-1265.21 Version 10.9.4 (Build 13E28))
>>> :  of 2014-09-02
>>> : Org-mode version 8.3beta (release_8.3beta-717-ga8096c)
>>> : ess-version: 14.05 [git: 4283f1304a54502c42707b6a4ba347703f0992dd]
>>>
>>
>> ,----
>> | #+RESULTS:
>> | : GNU Emacs 24.4.1 (x86_64-apple-darwin14.0.0, Carbon Version 157 AppKit 1343.16)
>> | :  of 2014-11-24
>> | : Org-mode version 8.3beta (release_8.3beta-695-g390015)
>> | : ess-version: 14.09 [svn: 6041 (2014-09-13)]
>> `----
>>
>>
>> I get the same in the preview, but it works for me. I do not use -q.
>>
>> ,----
>> | > data.table <- local({
>> | +      con <- textConnection(
>> | +        "\"A\"	\"B\"	\"C\"
>> | + \"115\"	\"76\"	\"60\"
>> | + \"124\"	\"78\"	\"55\"
>> | + \"118\"	\"73\"	\"65\"
>> | + \"114\"	\"75\"	\"61\"
>> | + \"108\"	\"74\"	\"82\""
>> | +      )
>> | +      res <- utils::read.table(
>> | +        con,
>> | +        header    = TRUE,
>> | +        row.names = NULL,
>> | +        sep       = "\t",
>> | +        as.is     = TRUE
>> | +      )
>> | +      close(con)
>> | +      res
>> | +    })
>> | >   names(data.table)
>> | [1] "A" "B" "C"
>> | >   head(data.table)
>> |     A  B  C
>> | 1 115 76 60
>> | 2 124 78 55
>> | 3 118 73 65
>> | 4 114 75 61
>> | 5 108 74 82
>> | > 
>> `----
>>
>>
>>>
>>> Here is what I see in the org babel preview:
>>>
>>
>> This might be due to temail, but you don't have tabs (\t) between your
>> entries below - could you verify if this is the case? If yes, this might
>> be a locale issue (would be strange)?
>
> Yes, there seem to be no tabs.  If there were tabs, R should parse the
> table correctly, I guess.

Exactly. So somehow the tabs get replaced in your system by spaces.

>
> (BTW: My custom filter replacing tabs with spaces is still disabled,
> even though I pulled from master, which includes the recent
>
> commit 70f965535b07c03a40792237de3a970d1cb57551
> Author: Nicolas Goaziou <mail@nicolasgoaziou.fr>
> Date:   Mon Jan 5 12:39:26 2015 +0100
>
>     org-table: Ignore filters and hooks in radio tables
> )
>
>
>>
>>> data.table <- local({
>>>      con <- textConnection(
>>>        "\"A\"   \"B\"   \"C\"
>>> \"115\" \"76\"  \"60\"
>>> \"124\" \"78\"  \"55\"
>>> \"118\" \"73\"  \"65\"
>>> \"114\" \"75\"  \"61\"
>>> \"108\" \"74\"  \"82\""
>>>      )
>>>      res <- utils::read.table(
>>>        con,
>>>        header    = TRUE,
>>>        row.names = NULL,
>>>        sep       = "\t",
>>>        as.is     = TRUE
>>>      )
>>>      close(con)
>>>      res
>>>    })
>>> names(data.table)
>>> head(data.table)
>>
>> One solution would be to insert (instead of the tab character) "\t" to
>> separate the fields?
>>
>> You could try and modify the ob-R-transfer-variable-table-with-header -
>> From the help:
>>
>> ,----
>> | ob-R-transfer-variable-table-with-header is a variable defined in `ob-R.el'.
>> | Its value is
>> | "%s <- local({\n con <- textConnection(\n %S\n )\n res <-
>> | utils::read.table(\n con,\n header = %s,\n row.names = %s,\n sep =
>> | \"\\t\",\n as.is = TRUE\n )\n close(con)\n res\n })"
>> | 
>> |   This variable may be risky if used as a file-local variable.
>> | 
>> | Documentation:
>> | R code used to transfer a table defined as a variable from org to R.
>> | 
>> | This function is used when the table contains a header.
>> `----
>>
>> This is passed to the (format) function and one might be able to change
>> something there.
>
> Thanks for the hint.  Do I assume correctly, that this variable is used
> when the code block is evaluated as well?  In that case, I am reluctant
> to change it, as the evaluation now works.  The surprising thing is the
> different behaviour for preview and evaluation.

Yup - you are right. These variables (ob-R-transfer-variable-table-with-header and
ob-R-transfer-variable-table-without-header) are only used in the
function org-babel-R-assign-elisp which is only called from
org-babel-variable-assignments:R which is doing the variable assignments
whenever they are needed.

What would definitely solve the issue is if the string containing the
tabs would use \t instead - but I have no idea how this could be
achieved easily.

>
> I am extremely tight with my time right now, so that I won't be able to
> do much experimentation/investigation right now, I am afraid.  But I
> will try to still be helpful...

The question is really "where does the tab become a space". As you see
the spaces after C-c C-v v it seems that the problem also occurs when
the function  org-babel-expand-src-block is called. So I think to narrow
down what is happening, you have to step through this function and see
if the assignments contain spaces instead of tabs from the beginning or
if they are changed whithin the function.

As only you see these errors, I would still guess that it is something
with your system (locale?, utf et al issues?).

By the way - are you starting R as --vanilla? If not, it might be worth
trying it out?

,----
|   (setq inferior-R-args "--vanilla")
`----

Actually, if you don't see tabs after calling org-babel-expand-src-block
on the example block, this is a pure org issue - no ESS or R involved here.

One thing you could try is to change the sep value in the variables
ob-R-transfer-variable-table-with-header and
ob-R-transfer-variable-table-without-header as follow:

,----
|   ...  
|        sep       = \"\",   ;;;;;<<<<<<<<< was before  sep = \"\\t\"
|   ...
`----

This will set the separator to "whitespace", i.e. "that is one or more
spaces, tabs, newlines or carriage returns" (from R help).

This should solve your problem.

Let us know if this solves your problem.

Cheers,

Rianer

>
> Thanks,
> Andreas
>
>
>>
>> Cheers,
>>
>> Rainer
>>
>>>
>>>
>>> And finally, here is my R session, when I step through the code:
>>>
>>>
>>> R version 3.1.2 (2014-10-31) -- "Pumpkin Helmet"
>>> Copyright (C) 2014 The R Foundation for Statistical Computing
>>> Platform: x86_64-apple-darwin10.8.0 (64-bit)
>>>
>>> R is free software and comes with ABSOLUTELY NO WARRANTY.
>>> You are welcome to redistribute it under certain conditions.
>>> Type 'license()' or 'licence()' for distribution details.
>>>
>>>   Natural language support but running in an English locale
>>>
>>> R is a collaborative project with many contributors.
>>> Type 'contributors()' for more information and
>>> 'citation()' on how to cite R or R packages in publications.
>>>
>>> Type 'demo()' for some demos, 'help()' for on-line help, or
>>> 'help.start()' for an HTML browser interface to help.
>>> Type 'q()' to quit R.
>>>
>>>> > options(STERM='iESS', str.dendrogram.last="'", editor='emacsclient', show.error.locations=TRUE)
>>>> data.table <- local({
>>> +      con <- textConnection(
>>> +        "\"A\"   \"B\"   \"C\"
>>> + \"115\" \"76\"  \"60\"
>>> + \"124\" \"78\"  \"55\"
>>> + \"118\" \"73\"  \"65\"
>>> + \"114\" \"75\"  \"61\"
>>> + \"108\" \"74\"  \"82\""
>>> +      )
>>> +      res <- utils::read.table(
>>> +        con,
>>> +        header    = TRUE,
>>> +        row.names = NULL,
>>> +        sep       = "\t",
>>> +        as.is     = TRUE
>>> +      )
>>> +      close(con)
>>> +      res
>>> +    })
>>>> names(data.table)
>>> [1] "A...B...C"
>>>> head(data.table)
>>>    A...B...C
>>> 1 115 76  60
>>> 2 124 78  55
>>> 3 118 73  65
>>> 4 114 75  61
>>> 5 108 74  82
>>>
>>>
>>> Regards,
>>> Andreas
>>>
>>>
>>>
>
>
>

-- 
Rainer M. Krug
email: Rainer<at>krugs<dot>de
PGP: 0x0F52F982

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 494 bytes --]

Re: org tables into R?

[Aaron Ecay]

Hi Rainer and Andreas,

2015ko urtarrilak 6an, Rainer M Krug-ek idatzi zuen:
> What would definitely solve the issue is if the string containing the
> tabs would use \t instead - but I have no idea how this could be
> achieved easily.

This could be achieved by modifying the call to orgtbl-to-csv in
org-babel-R-assign-elisp:

diff --git i/lisp/ob-R.el w/lisp/ob-R.el
index 6f76aa5..ec3f110 100644
--- i/lisp/ob-R.el
+++ w/lisp/ob-R.el
@@ -239,7 +239,10 @@ This function is called by `org-babel-execute-src-block'."
             (min (if lengths (apply 'min lengths) 0)))
         ;; Ensure VALUE has an orgtbl structure (depth of at least 2).
         (unless (listp (car value)) (setq value (list value)))
-       (let ((file (orgtbl-to-tsv value '(:fmt org-babel-R-quote-tsv-field)))
+       (let ((file (replace-regexp-in-string
+                    "\t" "\\t"
+                    (orgtbl-to-tsv value '(:raw t :fmt org-babel-R-quote-tsv-field))
+                    nil t))
              (header (if (or (eq (nth 1 value) 'hline) colnames-p)
                          "TRUE" "FALSE"))
              (row-names (if rownames-p "1" "NULL")))

(This is just a quick hack; if it works then the code should actually
use orgtbl-to-generic to do the conversion in a single step.)

> ,----
> |   ...  
> |        sep       = \"\",   ;;;;;<<<<<<<<< was before  sep = \"\\t\"
> |   ...
> `----
> 
> This will set the separator to "whitespace", i.e. "that is one or more
> spaces, tabs, newlines or carriage returns" (from R help).

This was the approach that Michael took in his original patch.  It
introduces new bugs related to the handling of quotes, though – see the
“Details” section of ?scan in R for the gory details (beginning with “If
‘sep’ is the default”...).

Andreas raised a bug a while ago that indicates to me that this code
should switch to csv rather than a whitespace-delimited format.  But
there are lots of corner cases, and I have not had time to work on a
patch that I am confident could cover them all.

Andreas’s original report (also referenced earlier in this thread):
http://news.gmane.org/find-root.php?message_id=olulhmmtei5.fsf%40med.uni%2dgoettingen.de

Thanks,

-- 
Aaron Ecay

Re: org tables into R?

[Nicolas Goaziou]

Aaron Ecay <aaronecay@gmail.com> writes:

> You are correct about the silent dropping of macro-like text.  However,
> with current master that case gives an undefined macro error, which is
> even worse.

I disagree. If we allow to insert macro-like text in the table, it will
become active in the generated table and will, sooner or later, generate
an undefined macro error anyway.

> Try this (in emacs -Q with org and ESS in the load-path) to
> see it:
>
> #+name: foo
> #+begin_src R
>   c("foo","{{{bar}}}")
> #+end_src

Yes, an error is thrown, but

  #+RESULTS: foo
  |    foo    |
  | {{{bar}}} |

is as cheesy. See above.

Macros are a very special kind of datum in Org. Luckily, their syntax is
awkward so you're unlikely to create one unwillingly. How common is your
example?

>> OTOH "ob-R.el" should consider using ":raw t" parameter for its table
>> conversion function.
>
> I think :raw is needed in ‘org-babel-insert-result’ in addition to my
> previous patch.  New patch attached.

Actually, my advice doesn't stand. ":raw t" is not sufficient to prevent
macro expansion, which happens at the beginning of the export process.


Regards,

Re: org tables into R?

[Aaron Ecay]

Hi Nicolas,

2015ko urtarrilak 6an, Nicolas Goaziou-ek idatzi zuen:
> 
> Aaron Ecay <aaronecay@gmail.com> writes:
> 
>> You are correct about the silent dropping of macro-like text.  However,
>> with current master that case gives an undefined macro error, which is
>> even worse.
> 
> I disagree. If we allow to insert macro-like text in the table, it will
> become active in the generated table and will, sooner or later, generate
> an undefined macro error anyway.
> 
>> Try this (in emacs -Q with org and ESS in the load-path) to
>> see it:
>> 
>> #+name: foo
>> #+begin_src R
>> c("foo","{{{bar}}}")
>> #+end_src
> 
> Yes, an error is thrown, but
> 
>   #+RESULTS: foo
>   |    foo    |
>   | {{{bar}}} |
> 
> is as cheesy. See above.
> 
> Macros are a very special kind of datum in Org. Luckily, their syntax is
> awkward so you're unlikely to create one unwillingly. How common is your
> example?

Nothing unwilling about it – I had been deliberately trying to generate a
table with macros in it as the result of a babel block.  These macros are
defined in the document, in order to encapsulate some fiddly typesetting
which recurs very commonly.

Shouldn’t this workflow be supported?

In any case, the point is broader: the orgtbl-* functions cannot cope
with macros in their input at all (not just in the context of babel).  I
think the programmatic interface to org tables ought to be composable
with macros.

Thanks,

-- 
Aaron Ecay

Re: org tables into R?

[Nicolas Goaziou]

Aaron Ecay <aaronecay@gmail.com> writes:

> Nothing unwilling about it – I had been deliberately trying to generate a
> table with macros in it as the result of a babel block.  These macros are
> defined in the document, in order to encapsulate some fiddly typesetting
> which recurs very commonly.
>
> Shouldn’t this workflow be supported?

Note that Babel blocks are evaluated after macro expansion. As
a consequence, the workflow above is hardly useful, as it requires its
user to manually evaluate all blocks generating macros /before/
exporting the document.

If the workflow is not used and doesn't have intrinsic value, I don't
think we need to invent kludges to support it.

> In any case, the point is broader: the orgtbl-* functions cannot cope
> with macros in their input at all (not just in the context of babel).  I
> think the programmatic interface to org tables ought to be composable
> with macros.

orgtbl-* are not an API for tables, but converters.

My point is still valid: you don't have a real use-case, so just don't
bother.


Regards,

Re: org tables into R?

[Rainer M Krug]

[-- Attachment #1: Type: text/plain, Size: 3258 bytes --]

Aaron Ecay <aaronecay@gmail.com> writes:

> Hi Rainer and Andreas,
>
> 2015ko urtarrilak 6an, Rainer M Krug-ek idatzi zuen:
>> What would definitely solve the issue is if the string containing the
>> tabs would use \t instead - but I have no idea how this could be
>> achieved easily.
>
> This could be achieved by modifying the call to orgtbl-to-csv in
> org-babel-R-assign-elisp:
>
> diff --git i/lisp/ob-R.el w/lisp/ob-R.el
> index 6f76aa5..ec3f110 100644
> --- i/lisp/ob-R.el
> +++ w/lisp/ob-R.el
> @@ -239,7 +239,10 @@ This function is called by `org-babel-execute-src-block'."
>              (min (if lengths (apply 'min lengths) 0)))
>          ;; Ensure VALUE has an orgtbl structure (depth of at least 2).
>          (unless (listp (car value)) (setq value (list value)))
> -       (let ((file (orgtbl-to-tsv value '(:fmt org-babel-R-quote-tsv-field)))
> +       (let ((file (replace-regexp-in-string
> +                    "\t" "\\t"
> +                    (orgtbl-to-tsv value '(:raw t :fmt org-babel-R-quote-tsv-field))
> +                    nil t))
>               (header (if (or (eq (nth 1 value) 'hline) colnames-p)
>                           "TRUE" "FALSE"))
>               (row-names (if rownames-p "1" "NULL")))
>
> (This is just a quick hack; if it works then the code should actually
> use orgtbl-to-generic to do the conversion in a single step.)

That looks like it would work. 

>
>> ,----
>> |   ...  
>> |        sep       = \"\",   ;;;;;<<<<<<<<< was before  sep = \"\\t\"
>> |   ...
>> `----
>> 
>> This will set the separator to "whitespace", i.e. "that is one or more
>> spaces, tabs, newlines or carriage returns" (from R help).
>
> This was the approach that Michael took in his original patch.  It
> introduces new bugs related to the handling of quotes, though – see the
> “Details” section of ?scan in R for the gory details (beginning with “If
> ‘sep’ is the default”...).

I agree completely - it would not be a final solution but possibly solve
the problem temporarily in an emacs.org file.

But the real question is: why are on some setups the tabs converted to
spaces?

>
> Andreas raised a bug a while ago that indicates to me that this code
> should switch to csv rather than a whitespace-delimited format.  But
> there are lots of corner cases, and I have not had time to work on a
> patch that I am confident could cover them all.

Might be a solution - but I do not know how commas are working under
different locales. In MS Office, a csv file is a semicolon (;)
separated file when in the chosen language settings a comma is used as
decimal separator - and and I have no idea if language settings affect
emacs in the same way (I actually don't hope so).

This makes me think - why not use "|" as a separator (as it is the case
in org)? In this way, all tables which are in org can be directly given
to R as strings without the hassle of to much of conversion?

Rainer

>
> Andreas’s original report (also referenced earlier in this thread):
> http://news.gmane.org/find-root.php?message_id=olulhmmtei5.fsf%40med.uni%2dgoettingen.de
>
> Thanks,

-- 
Rainer M. Krug
email: Rainer<at>krugs<dot>de
PGP: 0x0F52F982

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 494 bytes --]