[BUG] R src blocks untabify literal constant TABs

[BUG] R src blocks untabify literal constant TABs

[Charles C. Berry]

ECM follows:

Make a string of three characters :'A', '<TAB>', and 'B':

#+BEGIN_SRC R
res <- paste0("A","\t","B")
nchar(res)
#+END_SRC

#+RESULTS:
: 3

Make a src block in which the string appears literally, then execute the 
resulting src block:

#+BEGIN_SRC R  :results raw :wrap "src R"
   res <- paste0("A","\t","B")
   paste0("nchar(\"",res,"\")")
#+END_SRC

#+RESULTS:
#+BEGIN_src R
nchar("A	B")
#+END_src

#+RESULTS:
: 10

Ouch!

--

I don't have time to dig into this now, but thought I'd get it on the 
record.


Chuck

Re: [BUG] R src blocks untabify literal constant TABs

[Grant Rettke]

On Tue, Sep 22, 2015 at 5:04 PM, Charles C. Berry <ccberry@ucsd.edu> wrote:
>
> ECM follows:

I just ran it and got 10 chars like you expected:

#+BEGIN_src R
nchar("A        B")
#+END_src

#+RESULTS:
: 10

---

Emacs Version: src_emacs-lisp{emacs-version} {{{results("24.4.1")}}}
Org Version from Git: src_emacs-lisp{org-version} {{{results("8.3.1")}}}

Init file:

;; -*- lexical-binding: t -*-

(setq load-prefer-newer t)
(add-to-list 'load-path "~/src/org-mode/lisp")
(add-to-list 'load-path "~/src/org-mode/contrib/lisp")
(setq org-list-allow-alphabetical t)
(setq org-enforce-todo-checkbox-dependencies t)
(require 'org)
(org-babel-do-load-languages
 'org-babel-load-languages
 '((emacs-lisp . t)
   (org . t)
   ;;
   (C . t)
   (R . t)
   (python . t)
   (sass . t)
   (scheme . t)
   (sql . t)
   ;;
   (latex . t)
   ;;
   (makefile . t)
   (sh . t)
   ;;
   (ditaa . t)
   (dot . t)
   (plantuml . t)))

Re: [BUG] R src blocks untabify literal constant TABs

[Charles C. Berry]

On Tue, 22 Sep 2015, Grant Rettke wrote:

> On Tue, Sep 22, 2015 at 5:04 PM, Charles C. Berry <ccberry@ucsd.edu> wrote:
>>
>> ECM follows:
>
> I just ran it and got 10 chars like you expected:
>
> #+BEGIN_src R
> nchar("A        B")
> #+END_src
>
> #+RESULTS:
> : 10

10 is not what I expected. Should be 3 characters.

Be sure to run the second src block (which creates the third with the 
three character string) before running the third.

But thanks for checking.

Chuck

Re: [BUG] R src blocks untabify literal constant TABs

[Charles C. Berry]

Sorry, I just walked in the door and didn't put on my thinking cap...

On Tue, 22 Sep 2015, Charles C. Berry wrote:

> On Tue, 22 Sep 2015, Grant Rettke wrote:
>
>> On Tue, Sep 22, 2015 at 5:04 PM, Charles C. Berry <ccberry@ucsd.edu> wrote:
>>> 
>>> ECM follows:
>> 
>> I just ran it and got 10 chars like you expected:

I got 10, too. That is the problem. Maybe that is what you meant?

What I expected was 3 characters like the first src block gave. 
The reason the block just below fails is that some un-tab-ing puts spaces 
where a tab should be.

>> 
>> #+BEGIN_src R
>> nchar("A        B")
>> #+END_src
>> 
>> #+RESULTS:
>> : 10
>
> 10 is not what I expected. Should be 3 characters.
>
> Be sure to run the second src block (which creates the third with the three 
> character string) before running the third.
>

If you do run the second block, the third block should have A, <TAB>, and 
B in the quoted string. It does for me. And then when executed gives the 
erroneous 10.

> But thanks for checking.
>
> Chuck
>
>

Chuck

Re: [BUG] R src blocks untabify literal constant TABs

[Charles C. Berry]

On Tue, 22 Sep 2015, Charles C. Berry wrote:

Just to tie this up, this is an issue with `org-src-preserve-indentation', 
see

 	(info "(org) Editing source code")

for the best documentation. I found the "replace sequences of spaces with 
tab characters" for nil values confusing. The opposite is happening in 
this example.

Anyway this shows the behavior I expected. I guess that my cry of "BUG!" 
was unawarranted.

--8<---------------cut here---------------start------------->8---

#+BEGIN_SRC R  :results raw :wrap "src R -i"
    res <- paste0("A","\t","B")
    paste0("nchar(\"",res,"\")")
#+END_SRC

#+RESULTS:
#+BEGIN_src R -i
nchar("A	B")
#+END_src

#+RESULTS:
: 3

--8<---------------cut here---------------end--------------->8---

Chuck

Re: [BUG] R src blocks untabify literal constant TABs

[Nicolas Goaziou]

Hello,

"Charles C. Berry" <ccberry@ucsd.edu> writes:

> On Tue, 22 Sep 2015, Charles C. Berry wrote:
>
> Just to tie this up, this is an issue with `org-src-preserve-indentation', 
> see
>
>  	(info "(org) Editing source code")
>
> for the best documentation. I found the "replace sequences of spaces with 
> tab characters" for nil values confusing. The opposite is happening in 
> this example.

Would you want to write a documentation patch about it?

Regards,

-- 
Nicolas Goaziou

Re: [BUG] R src blocks untabify literal constant TABs

[Charles C. Berry]

On Thu, 24 Sep 2015, Nicolas Goaziou wrote:

> Hello,
>
> "Charles C. Berry" <ccberry@ucsd.edu> writes:
>
>> On Tue, 22 Sep 2015, Charles C. Berry wrote:
>>
>> Just to tie this up, this is an issue with `org-src-preserve-indentation',
>> see
>>
>>  	(info "(org) Editing source code")
>>
>> for the best documentation. I found the "replace sequences of spaces with
>> tab characters" for nil values confusing. The opposite is happening in
>> this example.
>
> Would you want to write a documentation patch about it?

I'll take a look.

There are about 70 places where

grep -i -nH -e preserve-indentation\\\|remove-indentation

matches in lisp and several different functions.

So I feel a need to be sure I get it all right.

Chuck

Re: [BUG] R src blocks untabify literal constant TABs

[Charles C. Berry]

[-- Attachment #1: Type: text/plain, Size: 1322 bytes --]

On Thu, 24 Sep 2015, Nicolas Goaziou wrote:

> Hello,
>
> "Charles C. Berry" <ccberry@ucsd.edu> writes:
>
>> On Tue, 22 Sep 2015, Charles C. Berry wrote:
>>
>> Just to tie this up, this is an issue with `org-src-preserve-indentation',
>> see
>>
>>  	(info "(org) Editing source code")
>>
>> for the best documentation. I found the "replace sequences of spaces with
>> tab characters" for nil values confusing. The opposite is happening in
>> this example.
>
> Would you want to write a documentation patch about it?
>

Attached is a patch.

I think it is an improvement, but `org-src-preserve-indentation' behaves 
differently according to context, which makes it hard to document 
succinctly.

In brief:

Setting to `t' or `nil' will keep or allow removal of tabs in these 
functions:

org-babel-parse-src-block-match
org-babel-haskell-export-to-lhs
org-babel-spec-to-string
org-src--edit-element
org-edit-src-code

but have no effect on embedded tabs in

org-babel-exp-process-buffer
org-export-unravel-code
org-element-src-block-interpreter
org-element-example-block-interpreter
org-fixup-indentation

Also, as briefly noted in the doc/org.texi part of the patch global 
indent/outdent operations (e.g. org-metaright on a heading) are affected.
This might make sense but sometimes the results surprise me.

HTH,

Chuck

[-- Attachment #2: org-src-preserve-indentation docs patch --]
[-- Type: text/plain, Size: 3700 bytes --]

From 12a912e9cbc1b2b475632dd724ca55af54b56236 Mon Sep 17 00:00:00 2001
From: Charles Berry <ccberry@ucsd.edu>
Date: Fri, 25 Sep 2015 12:02:32 -0700
Subject: [PATCH] org-src-preserve-indentation: docstring and info fixes

doc/org.texi: The `Literal examples' node mentions the effect of the
	      `-i' switch on embedded tabs.

	      The `Editing source code' node correctly states that
	      setting `org-src-preserve-indentation' to `t' prevents
	      converting tabs to spaces.  Also, its effect on
	      indentation operations is noted. A mention of the `-i'
	      switch is given.

lisp/org-src.el: The `org-src-preserve-indentation' docstring mentions
		 the effect of setting its value to `t' is to protect
		 conversion of tabs to spaces.
---
 doc/org.texi    | 17 ++++++++++-------
 lisp/org-src.el |  7 ++++---
 2 files changed, 14 insertions(+), 10 deletions(-)

diff --git a/doc/org.texi b/doc/org.texi
index 8de8efc..62c39e3 100644
--- a/doc/org.texi
+++ b/doc/org.texi
@@ -10053,8 +10053,8 @@ jumps to point-min.
 @end example
 
 @cindex indentation, in source blocks
-Finally, you can use @code{-i} to preserve the indentation of a specific code
-block (@pxref{Editing source code}).
+Finally, you can use @code{-i} to preserve the indentation and tab
+characters of a specific code block (@pxref{Editing source code}).
 
 @vindex org-coderef-label-format
 If the syntax for the label format conflicts with the language syntax, use a
@@ -14826,11 +14826,14 @@ Controls the way Emacs windows are rearranged when the edit buffer is created.
 @cindex indentation, in source blocks
 By default, the value is @code{nil}, which means that when code blocks are
 evaluated during export or tangled, they are re-inserted into the code block,
-which may replace sequences of spaces with tab characters.  When non-@code{nil},
-whitespace in code blocks will be preserved during export or tangling,
-exactly as it appears.  This variable is especially useful for tangling
-languages such as Python, in which whitespace indentation in the output is
-critical.
+which may replace tab characters with sequences of spaces.  When
+non-@code{nil}, whitespace in code blocks will be preserved during export or
+tangling, exactly as it appears.  This variable is especially useful for
+tangling languages such as Python, in which whitespace indentation in the
+output is critical.  A value of @code{t} prevents global indent/outdent
+operations from moving source or example block contents. The @code{-i} switch
+has a similar effect on a single source or example block (@pxref{Literal
+examples}).
 @item org-src-ask-before-returning-to-edit-buffer
 By default, Org will ask before returning to an open edit buffer.  Set this
 variable to @code{nil} to switch without asking.
diff --git a/lisp/org-src.el b/lisp/org-src.el
index 9c205e1..394633d 100644
--- a/lisp/org-src.el
+++ b/lisp/org-src.el
@@ -108,9 +108,10 @@ These are the regions where each line starts with a colon."
 
 (defcustom org-src-preserve-indentation nil
   "If non-nil preserve leading whitespace characters on export.
-If non-nil leading whitespace characters in source code blocks
-are preserved on export, and when switching between the org
-buffer and the language mode edit buffer.
+If non-nil leading whitespace characters and embedded tab
+characters in source code blocks are preserved on export, and
+when switching between the org buffer and the language mode edit
+buffer.
 
 When this variable is nil, after editing with \\[org-edit-src-code],
 the minimum (across-lines) number of leading whitespace characters
-- 
2.3.8 (Apple Git-58)

Re: [BUG] R src blocks untabify literal constant TABs

[Nicolas Goaziou]

Hello,

"Charles C. Berry" <ccberry@ucsd.edu> writes:

> Attached is a patch.

Thank you.

> I think it is an improvement, but `org-src-preserve-indentation'
> behaves differently according to context, which makes it hard to
> document succinctly.

There's something suspicious here.

This variable is supposed to only replace tab characters used in
indentation, not anywhere else. However, it seems that your
documentation patch suggests otherwise. It may be a bug then.

Do you have any such example?


Regards,

-- 
Nicolas Goaziou

Re: [BUG] R src blocks untabify literal constant TABs

[Charles C. Berry]

On Wed, 7 Oct 2015, Nicolas Goaziou wrote:

> Hello,
>
> "Charles C. Berry" <ccberry@ucsd.edu> writes:
>
>> Attached is a patch.
>
> Thank you.
>
>> I think it is an improvement, but `org-src-preserve-indentation'
>> behaves differently according to context, which makes it hard to
>> document succinctly.
>
> There's something suspicious here.
>
> This variable is supposed to only replace tab characters used in
> indentation, not anywhere else. However, it seems that your
> documentation patch suggests otherwise. It may be a bug then.
>

More than one bug, I am afraid.

> Do you have any such example?

Sure, make sure that the whitespace in the `> <' below is a TAB
character (C-q C-i). Running the src block replaces the TAB when
`org-src-preserve-indentation' is nil.

Also, use `org-edit-src-code' with point in the block and immediately exist,
i.e. =C-c ' C-c '=, and the embedded tab disappears.

#+BEGIN_SRC emacs-lisp
   "this is a TAB-->	<--"
   #+END_SRC

`untabify' is called by some functions that honor (in some way)
`org-src-preserve-indentation' which explains some of this. FWIW, here
is a summary of functions that use the variable:


|---------------------------------------+-----------------------+---------------------------|
| function                              | protect-leading space | effect on tabs            |
|---------------------------------------+-----------------------+---------------------------|
| org-babel-exp-process-buffer          | yes or no             | ignore (+/- to indent)    |
| org-export-unravel-code               | yes or no             | ignore                    |
| org-element-src-block-interpreter     | contents only or not  | ignore                    |
| org-element-example-block-interpreter | contents only or not  | ignore                    |
| org-fixup-indentation                 | contents only or not  | ignore (+/- to indent)    |
| org-babel-parse-src-block-match       | yes or no             | keep or remove            |
| org-babel-haskell-export-to-lhs       | yes or no             | keep or remove            |
| org-babel-spec-to-string              | yes or no             | keep or remove lead+trail |
| org-src--edit-element                 | yes or no             | keep or remove            |
| org-edit-src-code                     | yes or no             | keep or remove            |
|---------------------------------------+-----------------------+---------------------------|

the '(+/- to indent)' means that leading tabs may be added or removed
with org-metaright and friends. `keep or remove lead+trail' == "keep
or remove leading or trailing TABs only". 'contents only or not' means
there is no effect on `#+BEGIN...' etc.

HTH,

Chuck

Re: [BUG] R src blocks untabify literal constant TABs

[Nicolas Goaziou]

Hello,

"Charles C. Berry" <ccberry@ucsd.edu> writes:

> More than one bug, I am afraid.
>
>> Do you have any such example?
>
> Sure, make sure that the whitespace in the `> <' below is a TAB
> character (C-q C-i). Running the src block replaces the TAB when
> `org-src-preserve-indentation' is nil.
>
> Also, use `org-edit-src-code' with point in the block and immediately exist,
> i.e. =C-c ' C-c '=, and the embedded tab disappears.
>
> #+BEGIN_SRC emacs-lisp
>    "this is a TAB-->	<--"
>    #+END_SRC
>
> `untabify' is called by some functions that honor (in some way)
> `org-src-preserve-indentation' which explains some of this. FWIW, here
> is a summary of functions that use the variable:
>
>
> |---------------------------------------+-----------------------+---------------------------|
> | function                              | protect-leading space | effect on tabs            |
> |---------------------------------------+-----------------------+---------------------------|
> | org-babel-exp-process-buffer          | yes or no             | ignore (+/- to indent)    |
> | org-export-unravel-code               | yes or no             | ignore                    |
> | org-element-src-block-interpreter     | contents only or not  | ignore                    |
> | org-element-example-block-interpreter | contents only or not  | ignore                    |
> | org-fixup-indentation                 | contents only or not  | ignore (+/- to indent)    |
> | org-babel-parse-src-block-match       | yes or no             | keep or remove            |
> | org-babel-haskell-export-to-lhs       | yes or no             | keep or remove            |
> | org-babel-spec-to-string              | yes or no             | keep or remove lead+trail |
> | org-src--edit-element                 | yes or no             | keep or remove            |
> | org-edit-src-code                     | yes or no             | keep or remove            |
> |---------------------------------------+-----------------------+---------------------------|
>
> the '(+/- to indent)' means that leading tabs may be added or removed
> with org-metaright and friends. `keep or remove lead+trail' == "keep
> or remove leading or trailing TABs only". 'contents only or not' means
> there is no effect on `#+BEGIN...' etc.

The main culprit is `org-do-remove-indentation', which doesn't need to
untabify the buffer in order to proceed. I changed that in master. It
should go a long way towards fixing the issue.

Is there any case left to fix? If there isn't, is the current
documentation accurate enough or do you think it needs to be improved?


Regards,

-- 
Nicolas Goaziou

Re: [BUG] R src blocks untabify literal constant TABs

[Charles C. Berry]

On Thu, 8 Oct 2015, Nicolas Goaziou wrote:

> Hello,
>
> "Charles C. Berry" <ccberry@ucsd.edu> writes:
>
[...]
>
> The main culprit is `org-do-remove-indentation', which doesn't need to
> untabify the buffer in order to proceed. I changed that in master. It
> should go a long way towards fixing the issue.
>
> Is there any case left to fix? If there isn't, is the current
> documentation accurate enough or do you think it needs to be improved?
>

If `org-src-preserve-indentation' is nil, then embedded (not just 
leading/trailing) tabs in src blocks are converted to spaces when 
`org-edit-src-code' is called or when exported. Not sure I would call this 
a bug.

If that is not deemed a bug, the docstring needs to mention that embedded 
tabs get converted.

Apart from that, the documentation needs to reverse this in 14.2:

"may replace sequences of spaces with tab characters"

as it is spaces that replace tab characters.

Best,

Chuck

Re: [BUG] R src blocks untabify literal constant TABs

[Nicolas Goaziou]

"Charles C. Berry" <ccberry@ucsd.edu> writes:

> If `org-src-preserve-indentation' is nil, then embedded (not just
> leading/trailing) tabs in src blocks are converted to spaces when
> `org-edit-src-code' is called

This is now fixed in master. Thank you.

>or when exported.

I cannot reproduce it. Do you have an ECM ?

> Not sure I would call this a bug.

I think it is one.

> If that is not deemed a bug, the docstring needs to mention that
> embedded tabs get converted.
>
> Apart from that, the documentation needs to reverse this in 14.2:
>
> "may replace sequences of spaces with tab characters"
>
> as it is spaces that replace tab characters.

OK. So with the new behaviour, this is the only necessary change to
documentation?

Regards,

Re: [BUG] R src blocks untabify literal constant TABs

[Charles C. Berry]

On Thu, 8 Oct 2015, Nicolas Goaziou wrote:

> "Charles C. Berry" <ccberry@ucsd.edu> writes:
>
>> If `org-src-preserve-indentation' is nil, then embedded (not just
>> leading/trailing) tabs in src blocks are converted to spaces when
>> `org-edit-src-code' is called
>
> This is now fixed in master. Thank you.
>
>> or when exported.
>
> I cannot reproduce it. Do you have an ECM ?
>

I forgot `org-reload', I think. Seems AOK now.

>> Not sure I would call this a bug.
>
> I think it is one.
>
>> If that is not deemed a bug, the docstring needs to mention that
>> embedded tabs get converted.
>>
>> Apart from that, the documentation needs to reverse this in 14.2:
>>
>> "may replace sequences of spaces with tab characters"
>>
>> as it is spaces that replace tab characters.
>
> OK. So with the new behaviour, this is the only necessary change to
> documentation?

Yes.

Chuck

Re: [BUG] R src blocks untabify literal constant TABs

[Nicolas Goaziou]

"Charles C. Berry" <ccberry@ucsd.edu> writes:

>>> Apart from that, the documentation needs to reverse this in 14.2:
>>>
>>> "may replace sequences of spaces with tab characters"
>>>
>>> as it is spaces that replace tab characters.
>>
>> OK. So with the new behaviour, this is the only necessary change to
>> documentation?
>
> Yes.

Actually, I think neither is true because `indent-line-to' inserts tab
characters if indentation goes past `tab-width' and `indent-tabs-mode'
is non-nil.

What about the following:

  ‘org-src-preserve-indentation’ By default, the value is ‘nil’, which
       means that code blocks evaluated during export or tangled are
       indented according to context, possibly altering leading
       sequences of spaces and tab characters in the process. When
       non-nil, indentation is relative to left column, and therefore,
       not modified during export or tangling. This variable is
       especially useful for tangling languages such as Python, in which
       whitespace indentation in the output is critical.


Regards,

Re: [BUG] R src blocks untabify literal constant TABs

[Charles C. Berry]

[-- Attachment #1: Type: text/plain, Size: 664 bytes --]

On Sun, 11 Oct 2015, Nicolas Goaziou wrote:

>
> What about the following:
>
>  ‘org-src-preserve-indentation’ By default, the value is ‘nil’, which
>       means that code blocks evaluated during export or tangled are
>       indented according to context, possibly altering leading
>       sequences of spaces and tab characters in the process. When
>       non-nil, indentation is relative to left column, and therefore,
>       not modified during export or tangling. This variable is
>       especially useful for tangling languages such as Python, in which
>       whitespace indentation in the output is critical.
>


I think that is correct.

Chuck

Re: [BUG] R src blocks untabify literal constant TABs

[Nicolas Goaziou]

"Charles C. Berry" <ccberry@ucsd.edu> writes:

>> What about the following:
>>
>>  ‘org-src-preserve-indentation’ By default, the value is ‘nil’, which
>>       means that code blocks evaluated during export or tangled are
>>       indented according to context, possibly altering leading
>>       sequences of spaces and tab characters in the process. When
>>       non-nil, indentation is relative to left column, and therefore,
>>       not modified during export or tangling. This variable is
>>       especially useful for tangling languages such as Python, in which
>>       whitespace indentation in the output is critical.
>>
>
>
> I think that is correct.

Applied, then. Thank you.


Regards,