Strings converted to numbers in Org table?

Strings converted to numbers in Org table?

[Vicente Vera]

[-- Attachment #1: Type: text/plain, Size: 1293 bytes --]

Hello. I'm trying to get an Org table from an R data frame but data is
lost in the process.

Here is a MWE. Note that:

- In R every value is a string. "var2" contains no numbers (is a
  character vector).

- Upon conversion to a table Org removes the zero from "var2" last
  value.

------------------------------

#+BEGIN_SRC R :session *mwe* :results value table :colnames yes
  tst <- data.frame(var1 = c("a", "b", "c", "d", "e", "f", "g"),
                    var2 = c("150", "210", "140", "150", "192", "497",
"3.350"),
                    stringsAsFactors = FALSE)
  tst
#+END_SRC

#+RESULTS:
| var1 | var2 |
|------+------|
| a    |  150 |
| b    |  210 |
| c    |  140 |
| d    |  150 |
| e    |  192 |
| f    |  497 |
| g    | 3.35 |

------------------------------

Here's the output as seen in R:

: > tst
:   var1  var2
: 1    a   150
: 2    b   210
: 3    c   140
: 4    d   150
: 5    e   192
: 6    f   497
: 7    g 3.350

Details on the data frame:

: > str(tst)
: 'data.frame':    7 obs. of  2 variables:
:  $ var1: chr  "a" "b" "c" "d" ...
:  $ var2: chr  "150" "210" "140" "150" ...

It seems Org knows that the values on column "var2" are numbers and
converts the strings to numbers, applying some obscure trimming on the
digits. The "3.350" value needs to be left as is.

[-- Attachment #2: Type: text/html, Size: 1883 bytes --]

Re: Strings converted to numbers in Org table?

[Vicente Vera]

[-- Attachment #1: Type: text/plain, Size: 1753 bytes --]

Probably the issue is related to this function in `ob-core.el'?

org-babel-read: "Convert the string value of CELL to a number if
appropriate."

Behind the curtains lies the built-in function `string-to-number'.

Maybe that conversion should be made optional to leave the strings
untouched.

2017-02-24 1:40 GMT+00:00 Vicente Vera <vicentemvp@gmail.com>:

> Hello. I'm trying to get an Org table from an R data frame but data is
> lost in the process.
>
> Here is a MWE. Note that:
>
> - In R every value is a string. "var2" contains no numbers (is a
>   character vector).
>
> - Upon conversion to a table Org removes the zero from "var2" last
>   value.
>
> ------------------------------
>
> #+BEGIN_SRC R :session *mwe* :results value table :colnames yes
>   tst <- data.frame(var1 = c("a", "b", "c", "d", "e", "f", "g"),
>                     var2 = c("150", "210", "140", "150", "192", "497",
> "3.350"),
>                     stringsAsFactors = FALSE)
>   tst
> #+END_SRC
>
> #+RESULTS:
> | var1 | var2 |
> |------+------|
> | a    |  150 |
> | b    |  210 |
> | c    |  140 |
> | d    |  150 |
> | e    |  192 |
> | f    |  497 |
> | g    | 3.35 |
>
> ------------------------------
>
> Here's the output as seen in R:
>
> : > tst
> :   var1  var2
> : 1    a   150
> : 2    b   210
> : 3    c   140
> : 4    d   150
> : 5    e   192
> : 6    f   497
> : 7    g 3.350
>
> Details on the data frame:
>
> : > str(tst)
> : 'data.frame':    7 obs. of  2 variables:
> :  $ var1: chr  "a" "b" "c" "d" ...
> :  $ var2: chr  "150" "210" "140" "150" ...
>
> It seems Org knows that the values on column "var2" are numbers and
> converts the strings to numbers, applying some obscure trimming on the
> digits. The "3.350" value needs to be left as is.
>
>

[-- Attachment #2: Type: text/html, Size: 2580 bytes --]

Re: Strings converted to numbers in Org table?

[Vicente Vera]

[-- Attachment #1: Type: text/plain, Size: 2922 bytes --]

Hello again, I'm sorry for being noisy.

OK yes, `org-babel-read' is indeed converting "number strings" to
numbers.

Basically this is what happens:

: (string-to-number "3.350") => 3.35

To leave cell values unchanged I did this clumsy hack:

#+BEGIN_SRC emacs-lisp
  (defun test-other-org-babel-read (cell &optional inhibit-lisp-eval)
    (if (and (stringp cell) (not (equal cell "")))
        ;; ;; (or (org-babel--string-to-number cell)
        (if (and (not inhibit-lisp-eval)
                 (or (member (substring cell 0 1) '("(" "'" "`" "["))
                     (string= cell "*this*")))
            (eval (read cell) t)
          (if (string= (substring cell 0 1) "\"")
              (read cell)
            (progn (set-text-properties 0 (length cell) nil cell) cell)))
      ;; ;; )
      cell))

  ;; Override default behavior
  (fset 'org-babel-read 'test-other-org-babel-read)
#+END_SRC

It would be useful to have a header argument to prevent this
conversion. Probably somebody else has had the same issue?

2017-02-27 13:49 GMT+00:00 Vicente Vera <vicentemvp@gmail.com>:

> Probably the issue is related to this function in `ob-core.el'?
>
> org-babel-read: "Convert the string value of CELL to a number if
> appropriate."
>
> Behind the curtains lies the built-in function `string-to-number'.
>
> Maybe that conversion should be made optional to leave the strings
> untouched.
>
> 2017-02-24 1:40 GMT+00:00 Vicente Vera <vicentemvp@gmail.com>:
>
>> Hello. I'm trying to get an Org table from an R data frame but data is
>> lost in the process.
>>
>> Here is a MWE. Note that:
>>
>> - In R every value is a string. "var2" contains no numbers (is a
>>   character vector).
>>
>> - Upon conversion to a table Org removes the zero from "var2" last
>>   value.
>>
>> ------------------------------
>>
>> #+BEGIN_SRC R :session *mwe* :results value table :colnames yes
>>   tst <- data.frame(var1 = c("a", "b", "c", "d", "e", "f", "g"),
>>                     var2 = c("150", "210", "140", "150", "192", "497",
>> "3.350"),
>>                     stringsAsFactors = FALSE)
>>   tst
>> #+END_SRC
>>
>> #+RESULTS:
>> | var1 | var2 |
>> |------+------|
>> | a    |  150 |
>> | b    |  210 |
>> | c    |  140 |
>> | d    |  150 |
>> | e    |  192 |
>> | f    |  497 |
>> | g    | 3.35 |
>>
>> ------------------------------
>>
>> Here's the output as seen in R:
>>
>> : > tst
>> :   var1  var2
>> : 1    a   150
>> : 2    b   210
>> : 3    c   140
>> : 4    d   150
>> : 5    e   192
>> : 6    f   497
>> : 7    g 3.350
>>
>> Details on the data frame:
>>
>> : > str(tst)
>> : 'data.frame':    7 obs. of  2 variables:
>> :  $ var1: chr  "a" "b" "c" "d" ...
>> :  $ var2: chr  "150" "210" "140" "150" ...
>>
>> It seems Org knows that the values on column "var2" are numbers and
>> converts the strings to numbers, applying some obscure trimming on the
>> digits. The "3.350" value needs to be left as is.
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 4343 bytes --]

Re: Strings converted to numbers in Org table?

[Nicolas Goaziou]

Hello,

Vicente Vera <vicentemvp@gmail.com> writes:

> It would be useful to have a header argument to prevent this
> conversion. Probably somebody else has had the same issue?

Since this case may be rare, what about inserting a non breaking
zero-width space right before 3.350?

Regards,

-- 
Nicolas Goaziou

Re: Strings converted to numbers in Org table?

[Vicente Vera]

[-- Attachment #1: Type: text/plain, Size: 994 bytes --]

Yes, it is a rare case indeed. It's not necessary nor appropiate to
change Babel's behavior.

Long story short, I'm building several LaTeX tables from CSV files
which in turn come from Excel files. I'm not the author of these Excel
files and I have to reproduce them as faithful as possible.

The problem is that a comma is used as a decimal mark instead of a
dot. `string-to-number' correctly interprets numbers such as "1.100"
as floating point. But in this particular case this is not the
expected behavior and the tables get messed up.

Your solution is a much better option. Thanks.

2017-02-27 17:33 GMT+00:00 Nicolas Goaziou <mail@nicolasgoaziou.fr>:

> Hello,
>
> Vicente Vera <vicentemvp@gmail.com> writes:
>
> > It would be useful to have a header argument to prevent this
> > conversion. Probably somebody else has had the same issue?
>
> Since this case may be rare, what about inserting a non breaking
> zero-width space right before 3.350?
>
> Regards,
>
> --
> Nicolas Goaziou
>

[-- Attachment #2: Type: text/html, Size: 1539 bytes --]

Re: Strings converted to numbers in Org table?

[Charles C. Berry]

On Mon, 27 Feb 2017, Vicente Vera wrote:

> Hello again, I'm sorry for being noisy.
>
> OK yes, `org-babel-read' is indeed converting "number strings" to
> numbers.
>

[proposed fix deleted]

I don't have a feeling as to whether the proposed fix is appropriate.

However, there are other ways to solve this issue. See below.

>>>
>>> - In R every value is a string. "var2" contains no numbers (is a
>>>   character vector).
>>>
>>> - Upon conversion to a table Org removes the zero from "var2" last
>>>   value.
>>>
>>> ------------------------------
>>>
>>> #+BEGIN_SRC R :session *mwe* :results value table :colnames yes
>>>   tst <- data.frame(var1 = c("a", "b", "c", "d", "e", "f", "g"),
>>>                     var2 = c("150", "210", "140", "150", "192", "497",
>>> "3.350"),
>>>                     stringsAsFactors = FALSE)
>>>   tst
>>> #+END_SRC
>>>
>>> #+RESULTS:
>>> | var1 | var2 |
>>> |------+------|
>>> | a    |  150 |
>>> | b    |  210 |
>>> | c    |  140 |
>>> | d    |  150 |
>>> | e    |  192 |
>>> | f    |  497 |
>>> | g    | 3.35 |
>>>
>>> ------------------------------
>>>
>>> Here's the output as seen in R:
>>>
>>> : > tst
>>> :   var1  var2
>>> : 1    a   150
>>> : 2    b   210
>>> : 3    c   140
>>> : 4    d   150
>>> : 5    e   192
>>> : 6    f   497
>>> : 7    g 3.350
>>>
>>> Details on the data frame:
>>>
>>> : > str(tst)
>>> : 'data.frame':    7 obs. of  2 variables:
>>> :  $ var1: chr  "a" "b" "c" "d" ...
>>> :  $ var2: chr  "150" "210" "140" "150" ...
>>>
>>> It seems Org knows that the values on column "var2" are numbers and
>>> converts the strings to numbers, applying some obscure trimming on the
>>> digits. The "3.350" value needs to be left as is.


IMHO, it is often best to handle formatting of output in the language of 
the src block.  There are some tools for doing this in R: the `ascii' 
package is one. `xtable' is another.

You can always do something like this:

#+BEGIN_SRC R :session *mwe* :results output
  tst <- data.frame(var1 = c("a", "b", "c", "d", "e", "f", "g"),
  var2 = c("150", "210", "140", "150", "192", "497", "3.350"),
  stringsAsFactors = FALSE)

  cat( capture.output( print(tst, row.names=FALSE)), sep="\n")

#+END_SRC


Or you can use `ox-ravel'[1] to convert to one of the knitr-like formats 
and let knitr or rmarkdown handle the output conversion.

Best,

Chuck

[1] https://github.com/chasberry/orgmode-accessories

Re: Strings converted to numbers in Org table?

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 1051 bytes --]

On 2017-02-27 10:17, "Charles C. Berry" <ccberry@ucsd.edu> writes:

> IMHO, it is often best to handle formatting of output in the language of 
> the src block.  There are some tools for doing this in R: the `ascii' 
> package is one. `xtable' is another.

I've had the same problem when working with tables and ocaml, as the
latter requires everything in the table to have the same type, so I need
to make sure everything is a string. To work around this, I did the
following:

#+name: convert-table
#+BEGIN_SRC emacs-lisp :var data=tps
  (mapcar (lambda (row)
            (mapcar (lambda (cell)
                      (if
                          (numberp cell)
                          (number-to-string cell)
                        cell))
                    row))
          data)
#+END_SRC

#+BEGIN_SRC ocaml :var data=convert-table :results output raw
  (* ocaml code *)
#+END_SRC

Best,

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7
Monthly Athmospheric CO₂, Mauna Loa Obs. 2017-01: 406.13, 2016-01: 402.52

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 454 bytes --]