noweb and :var statements

noweb and :var statements

[Ken Mankoff]

Hi Org list,

I'm having with noweb and variables. Can someone explain what I'm doing wrong? For example, if I have this table:

#+NAME: table_foo
| foo |
|-----|
|  42 |
| 100 |

And I want to import it into Python and use it, I can do that like this:

#+NAME: import
#+BEGIN_SRC python :var table=table_foo :session foo
import numpy as np
table = np.array(table).astype(np.float).flatten()
#+END_SRC

Eval of this block works, and if I tangle it, I see:

> table=[[42], [100]]
> import numpy as np
> table = np.array(table).astype(np.float).flatten()

But if I want to use that block elsewhere via noweb, it doesn't seem to work:

#+BEGIN_SRC python :results output drawer :noweb yes :session foo :tangle import_noweb.py
<<import()>>
#+END_SRC

The code runs and in a clean *foo* session I do have my table variable, but I *also* get an error. The buffer contains the text at the bottom of this message, and the tangled code in import_noweb.py is only "nil".

How can I 1) run noweb blocks with variables and 2) tangle noweb blocks with variables (i.e. tangle tables into source files).

Thanks,

  -k.


table=[[42], [100]]
Python 2.7.15+ (default, Jul  9 2019, 16:51:35) 
[GCC 7.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> >>> python.el: native completion setup loaded
>>> 
>>> import numpy as np
table = np.array(table).astype(np.float).flatten()

open('/tmp/babel-Mb4ojd/python-GEnZeZ', 'w').write(str(_))


'org_babel_python_eoe'
>>> Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name '_' is not defined
>>> >>> >>> 'org_babel_python_eoe'
>>> 
>>> 
>>> 
>>> 
>>> 'org_babel_python_eoe'
'org_babel_python_eoe'
>>> 

Re: noweb and :var statements

[Sebastian Miele]

Hi Ken!

Ken Mankoff <mankoff@gmail.com> writes:

> I'm having with noweb and variables. Can someone explain what I'm
> doing wrong? For example, if I have this table:
>
> #+NAME: table_foo
> | foo |
> |-----|
> |  42 |
> | 100 |
>
> And I want to import it into Python and use it, I can do that like this:
>
> #+NAME: import
> #+BEGIN_SRC python :var table=table_foo :session foo
> import numpy as np
> table = np.array(table).astype(np.float).flatten()
> #+END_SRC
>
> Eval of this block works, and if I tangle it, I see:
>
>> table=[[42], [100]]
>> import numpy as np
>> table = np.array(table).astype(np.float).flatten()
>
> But if I want to use that block elsewhere via noweb, it doesn't seem to work:
>
> #+BEGIN_SRC python :results output drawer :noweb yes :session foo :tangle import_noweb.py
> <<import()>>
> #+END_SRC
>
> The code runs and in a clean *foo* session I do have my table
> variable, but I *also* get an error. The buffer contains the text at
> the bottom of this message, and the tangled code in import_noweb.py is
> only "nil".
>
> How can I 1) run noweb blocks with variables and 2) tangle noweb
> blocks with variables (i.e. tangle tables into source files).

Section 15.10 (Noweb Reference Syntax) of the manual says that e.g.
<<NAME(var=value)>> (a noweb reference with parentheses) executes the
NAMEd block with the specified binding and inserts the results of the
execution during tangling. That probably is why the table variable is
there in the session. But the results of executing <<import()>> at that
place were not what was intended.

If it were something like

#+NAME: import
#+BEGIN_SRC python :var table=table_foo :results output
import numpy as np
table = np.array(table).astype(np.float).flatten()
CODE
#+END_SRC

where CODE is replaced by some Python code that prints the wanted Python
code, it would work. (Note the omission of the :session and the addition
of :results output. See section 15.5 (Results of Evaluation) of the
manual for the reasons.)

As far as I understand the manual there may be no way to directly attain
what you are trying.

However, something like the following may suit your use case. (For the
header-args property see section 15.2 (Using Header Arguments) of the
manual.)

* A Heading
:PROPERTIES:
:header-args: :var table=table_foo
:END:

#+NAME: table_foo
| foo |
|-----|
|  42 |
| 100 |

#+NAME: import
#+BEGIN_SRC python
import numpy as np
table = np.array(table).astype(np.float).flatten()
#+END_SRC

#+BEGIN_SRC python :noweb yes :tangle import_noweb.py
<<import>>
#+END_SRC

Best wishes
Sebastian

Re: noweb and :var statements

[Sebastian Miele]

I wrote:

> [..]
>
> However, something like the following may suit your use case. (For the
> header-args property see section 15.2 (Using Header Arguments) of the
> manual.)
>
> * A Heading
> :PROPERTIES:
> :header-args: :var table=table_foo
> :END:
>
> #+NAME: table_foo
> | foo |
> |-----|
> |  42 |
> | 100 |
>
> #+NAME: import
> #+BEGIN_SRC python
> import numpy as np
> table = np.array(table).astype(np.float).flatten()
> #+END_SRC
>
> #+BEGIN_SRC python :noweb yes :tangle import_noweb.py
> <<import>>
> #+END_SRC

Just the following works too, of course:

#+NAME: table_foo
| foo |
|-----|
|  42 |
| 100 |

#+NAME: import
#+BEGIN_SRC python
import numpy as np
table = np.array(table).astype(np.float).flatten()
#+END_SRC

#+BEGIN_SRC python :noweb yes :var table=table_foo :tangle import_noweb.py
<<import>>
#+END_SRC

Re: noweb and :var statements

[Ken Mankoff]

Hi Sebastian,

I'm not getting the results I expect from your MWE either. Perhaps I gave too much code and asked X when what I really want is Y. I think I've distilled it to this:

What is the most elegant Org way to get a table into a Python array?

I can code it directly:

#+BEGIN_SRC python
<<setup>>
print(foo)
#+END_SRC


And now I can hide <<setup>> in a section at the bottom of the document. If it looks like this, everything works:

#+NAME: setup
#+BEGIN_SRC python
foo = np.array([42,43,44])
#+END_SRC

But is there a more elegant method? Can I get the same behavior if the data I want is in an Org table rather than hard-coded directly in Python?





Ideally, I'd like to have:

#+NAME: setup
#+BEGIN_SRC python
<<setup(table="foo_data" varname="foo")>>
<<setup(table="bar_data" varname="bar")>>
#+END_SRC

And a #+NAME: setup block that takes a :var table and sticks it in the :var varname variable.

And then after calling <<setup>> be able to use variable "foo" and "bar" that are generated from column or 2D Org tables elsewhere in the document. Can I do this in Org?

Thanks,

  -k.

Re: noweb and :var statements

[Ken Mankoff]

On 2019-10-06 at 21:52 +02, Sebastian Miele <sebastian.miele@gmail.com> wrote...
> I wrote:
>
>> [..]
>>
>> However, something like the following may suit your use case. (For the
>> header-args property see section 15.2 (Using Header Arguments) of the
>> manual.)
>>
>> * A Heading
>> :PROPERTIES:
>> :header-args: :var table=table_foo
>> :END:
>>
>> #+NAME: table_foo
>> | foo |
>> |-----|
>> |  42 |
>> | 100 |
>>
>> #+NAME: import
>> #+BEGIN_SRC python
>> import numpy as np
>> table = np.array(table).astype(np.float).flatten()
>> #+END_SRC
>>
>> #+BEGIN_SRC python :noweb yes :tangle import_noweb.py
>> <<import>>
>> #+END_SRC
>
> Just the following works too, of course:
>
> #+NAME: table_foo
> | foo |
> |-----|
> |  42 |
> | 100 |
>
> #+NAME: import
> #+BEGIN_SRC python
> import numpy as np
> table = np.array(table).astype(np.float).flatten()
> #+END_SRC
>
> #+BEGIN_SRC python :noweb yes :var table=table_foo :tangle import_noweb.py
> <<import>>
> #+END_SRC

The result of tangling this is "import_noweb.py" contains:

import numpy as np
table = np.array(table).astype(np.float).flatten()

And I'd like it to somewhere include the line:
table = [42,100]
or something similar. I don't have the table data in the tangled code.

  -k.

Re: noweb and :var statements

[Sebastian Miele]

Ken Mankoff <mankoff@gmail.com> writes:

> Hi Sebastian,
>
> I'm not getting the results I expect from your MWE either. Perhaps I
> gave too much code and asked X when what I really want is Y. I think
> I've distilled it to this:
>
> What is the most elegant Org way to get a table into a Python array?

I do not know whether it is the most elegant way, but the following
works, and probably is at least close to shortes possible way:

#+NAME: table_foo
| foo |
|-----|
|  42 |
| 100 |

#+BEGIN_SRC python :var table=table_foo :tangle table.py
#+END_SRC

After tangling, table.py contains "table=[[42], [100]]".

Did you know that you can have several source code blocks tangling to
the same file? Maybe you do not really need noweb stuff at all. If you
add a second block, e.g.

#+BEGIN_SRC python :tangle table.py
mangle_table
#+END_SRC

then, after tangling, table.py contains "table=[[42], [100]]" and
"mangle_table".


> I can code it directly:
>
> #+BEGIN_SRC python
> <<setup>>
> print(foo)
> #+END_SRC
>
>
> And now I can hide <<setup>> in a section at the bottom of the
> document. If it looks like this, everything works:
>
> #+NAME: setup
> #+BEGIN_SRC python
> foo = np.array([42,43,44])
> #+END_SRC
>
> But is there a more elegant method? Can I get the same behavior if the
> data I want is in an Org table rather than hard-coded directly in
> Python?

This use-case is covered by what I wrote above. You can put the named
Org table at the bottom of the file.

Another note: You can tangle to different files from one Org file. You
may e.g. add an additional source block:

#+BEGIN_SRC python :tangle use_table.py
include_table_py
use_table
#+END_SRC

where include_table_py must be replaced by the Python way of saying:
include the file table.py. (I do not know Python.)

> Ideally, I'd like to have:
>
> #+NAME: setup
> #+BEGIN_SRC python
> <<setup(table="foo_data" varname="foo")>>
> <<setup(table="bar_data" varname="bar")>>
> #+END_SRC
>
> And a #+NAME: setup block that takes a :var table and sticks it in the
> :var varname variable.

I strongly suspect that you do not really need noweb stuff and still
struggle to understand the very basics of tangling.

> And then after calling <<setup>> be able to use variable "foo" and
> "bar" that are generated from column or 2D Org tables elsewhere in the
> document. Can I do this in Org?
>
> Thanks,
>
>   -k.

Re: noweb and :var statements

[Sebastian Miele]

Ken Mankoff <mankoff@gmail.com> writes:

> On 2019-10-06 at 21:52 +02, Sebastian Miele <sebastian.miele@gmail.com> wrote...
>> I wrote:
>>
>>> [..]
>>>
>>> However, something like the following may suit your use case. (For the
>>> header-args property see section 15.2 (Using Header Arguments) of the
>>> manual.)
>>>
>>> * A Heading
>>> :PROPERTIES:
>>> :header-args: :var table=table_foo
>>> :END:
>>>
>>> #+NAME: table_foo
>>> | foo |
>>> |-----|
>>> |  42 |
>>> | 100 |
>>>
>>> #+NAME: import
>>> #+BEGIN_SRC python
>>> import numpy as np
>>> table = np.array(table).astype(np.float).flatten()
>>> #+END_SRC
>>>
>>> #+BEGIN_SRC python :noweb yes :tangle import_noweb.py
>>> <<import>>
>>> #+END_SRC
>>
>> Just the following works too, of course:
>>
>> #+NAME: table_foo
>> | foo |
>> |-----|
>> |  42 |
>> | 100 |
>>
>> #+NAME: import
>> #+BEGIN_SRC python
>> import numpy as np
>> table = np.array(table).astype(np.float).flatten()
>> #+END_SRC
>>
>> #+BEGIN_SRC python :noweb yes :var table=table_foo :tangle import_noweb.py
>> <<import>>
>> #+END_SRC
>
> The result of tangling this is "import_noweb.py" contains:
>
> import numpy as np
> table = np.array(table).astype(np.float).flatten()
>
> And I'd like it to somewhere include the line:
> table = [42,100]
> or something similar. I don't have the table data in the tangled code.

Strange. On my system, typing C-c C-v t (org-babel-tangle) in an Org
file with the above mentioned contents does yield a file import_noweb.py
with the following contents:

table=[[42], [100]]
import numpy as np
table = np.array(table).astype(np.float).flatten()

Please try it with emacs -Q. Maybe your config is broken.

Re: noweb and :var statements

[Sebastian Miele]

I wrote:

> [..]
>
> Please try it with emacs -Q. Maybe your config is broken.

After starting emacs -Q you will have to
M-x customize-variable RET org-babel-load-languages
and add Python as a loaded language.

Re: noweb and :var statements

[Ken Mankoff]

Hi Sebastian,

Thanks for your help. I was running with "-Q" but must have been making some other mistakes. It does work.

As for your other email... I do know several tangles can go to the same file. And I may be using <<noweb>> incorrectly, but I'm using it for the following reasons:

1) I'd like to bury code that must run first but is inconsequential at the bottom of the Org file. Noweb lets me have a tangled <<setup>> at the top, and then hide the lengthy <<setup>> code elsewhere. Is this a correct use case?

2) I'd like to import 10 tables, so I thought a noweb function might be useful for code reuse.

I finally got the behavior I'm looking for. What I need to remember/understand is that <<noweb>> just pastes the body, and <<noweb()>> evaluates the function. From this, my Python code needs to generate Python code! I now have the following MWE that behaves as I want both for in-buffer C-c C-c eval of main code block and tangled results. The key bit of code is the last babel block.

Thanks for your help,

  -k.


* MWE init
#+BEGIN_SRC emacs-lisp :results output
(setq org-confirm-babel-evaluate nil)
(org-babel-do-load-languages 'org-babel-load-languages '((python . t)))
(print (emacs-version))
(print (org-version nil t))
(org-babel-tangle)
#+END_SRC

* Main Project

#+NAME: main
#+BEGIN_SRC python :tangle MWE3.py :noweb yes :results output
<<setup>>
print(t42.sum())
print(t100.sum())
#+END_SRC

#+RESULTS: main
: 84.0
: 100.0

* Data Tables
#+NAME: table_42
| foo |
|-----|
|  42 |
|  42 |

#+NAME: table_100
| bar |
|-----|
| 100 |

* Setup

#+NAME: setup
#+BEGIN_SRC python :noweb yes
import numpy as np
<<import_table_to_varname(table=table_42, varname="t42")>>
<<import_table_to_varname(table=table_100, varname="t100")>>
#+END_SRC

* Table Import Code

#+NAME: import_table_to_varname
#+BEGIN_SRC python :var table=table_42 :var varname="foo" :noweb yes :results output
print(varname + "=np.array(" + str(table) + ").astype(np.float).flatten()")
#+END_SRC

Re: noweb and :var statements

[Sebastian Miele]

Ken Mankoff <mankoff@gmail.com> writes:

> Hi Sebastian,
>
> Thanks for your help. I was running with "-Q" but must have been
> making some other mistakes. It does work.
>
> As for your other email... I do know several tangles can go to the
> same file. And I may be using <<noweb>> incorrectly, but I'm using it
> for the following reasons:
>
> 1) I'd like to bury code that must run first but is inconsequential at
> the bottom of the Org file. Noweb lets me have a tangled <<setup>> at
> the top, and then hide the lengthy <<setup>> code elsewhere. Is this a
> correct use case?

Yes.

> 2) I'd like to import 10 tables, so I thought a noweb function might
> be useful for code reuse.
>
> I finally got the behavior I'm looking for. What I need to
> remember/understand is that <<noweb>> just pastes the body, and
> <<noweb()>> evaluates the function. From this, my Python code needs to
> generate Python code! I now have the following MWE that behaves as I
> want both for in-buffer C-c C-c eval of main code block and tangled
> results. The key bit of code is the last babel block.

A solution without having to write code-writing code is tangling to
different files (MWE3.py and setup.py). See the example below.

I read somewhere that Python has stong reflection features. It should be
possible to write Python code that, given a string and a value,
introduces a Python variable of that name and binds the value to it. If
you find out how that works, the mangle block below can be changed so
that the main block has e.g. setup.A.sum() instead of
setup.tables["A"].sum().

Final disclaimer: There may and probably are other possibilies that I do
not know or have not thought of.

* Main Project

#+NAME: main
#+BEGIN_SRC python :tangle MWE3.py :noweb yes :results output
import setup
print(setup.tables["A"].sum())
print(setup.tables["B"].sum())
#+END_SRC

* Data Tables
#+NAME: table_42
| foo |
|-----|
|  42 |
|  42 |

#+NAME: table_100
| bar |
|-----|
| 100 |

* Setup

#+BEGIN_SRC python :tangle setup.py
import numpy as np
tables={}
#+END_SRC

#+NAME: mangle
#+BEGIN_SRC python
tables[name] = np.array(table).astype(np.float).flatten()
#+END_SRC

#+BEGIN_SRC python :tangle setup.py :noweb yes :var table=table_42 :var name="A"
<<mangle>>
#+END_SRC

#+BEGIN_SRC python :tangle setup.py :noweb yes :var table=table_100 :var name="B"
<<mangle>>
#+END_SRC