Gather properties for use by babel source block?

Gather properties for use by babel source block?

[Tim Burt]

I want to gather data from properties into something that can be used by
a babel source block (e.g. plot the data).  Searches in the manual,
worg, and gmane have not yielded the method, but my best guess is that
I've missed it.  If so, this is simply a request for a pointer to the
documentation I should read.

BACKGROUND: I'm trying to complete a workflow where I capture data over
time and periodically operate with it (e.g. put into a database and
plot).  The excellent documentation for orgmode has made certain parts
easy going, but the current obstacle is extracting properties that can
subsequently be operated on in babel source blocks.
 + [X] Capture data into properties using templates
 + [X] View tables of properties in org file
   * [X] use columnview dynamic block
   * [X] use org-collector
 + [ ] Gather properties for use by babel source block
 + [-] Operate on the gathered properties 
   * [-] SQL or SQLite
     - [X] follow simple tutorials for proof of principle
     - [ ] populate database with gathered properties
   * [-] R
     - [X] follow simple tutorials
     - [ ] calculate statistics and populate database
     - [ ] plot from database
     - [ ] plot from table in org file

Good evening,
Tim

Re: Gather properties for use by babel source block?

[darcamo]

I have not used it myself, but isn't org-collector exactly what you want [1].
"org-collector collects headline properties into tables with optional
pre-processing"

Once you have a table, you can easily pass it to code blocks for further
processing with org-babel.

[1] - http://orgmode.org/worg/org-contrib/org-collector.html

--
Darlan

2011/8/2 Tim Burt <tcburt@rochester.rr.com>:
>
> I want to gather data from properties into something that can be used by
> a babel source block (e.g. plot the data).  Searches in the manual,
> worg, and gmane have not yielded the method, but my best guess is that
> I've missed it.  If so, this is simply a request for a pointer to the
> documentation I should read.
>
> BACKGROUND: I'm trying to complete a workflow where I capture data over
> time and periodically operate with it (e.g. put into a database and
> plot).  The excellent documentation for orgmode has made certain parts
> easy going, but the current obstacle is extracting properties that can
> subsequently be operated on in babel source blocks.
>  + [X] Capture data into properties using templates
>  + [X] View tables of properties in org file
>   * [X] use columnview dynamic block
>   * [X] use org-collector
>  + [ ] Gather properties for use by babel source block
>  + [-] Operate on the gathered properties
>   * [-] SQL or SQLite
>     - [X] follow simple tutorials for proof of principle
>     - [ ] populate database with gathered properties
>   * [-] R
>     - [X] follow simple tutorials
>     - [ ] calculate statistics and populate database
>     - [ ] plot from database
>     - [ ] plot from table in org file
>
> Good evening,
> Tim
>
>

Re: Gather properties for use by babel source block?

[tcburt]

Hello Darlan ---

---- "darcamo@gmail.com" <darcamo@gmail.com> wrote: 
> I have not used it myself, but isn't org-collector exactly what you want [1].
> "org-collector collects headline properties into tables with optional
> pre-processing"

Indeed org-collector has allowed me to extract properties into a table, and even without much knowledge of lisp (for the conditions) the results were promising.

> Once you have a table, you can easily pass it to code blocks for further
> processing with org-babel.

While I have the table from org-collector I do not know how to "easily pass it to code blocks".  I have an (untested) idea that if #+tblname preceded the table from org-collector, then a babel source block could operate on it.  If this works then almost all is well and good, except for the small detail of getting the #+tblname automatically from the org-collector process. 

Thank you for the response,
Tim
> 
> [1] - http://orgmode.org/worg/org-contrib/org-collector.html
> 
> --
> Darlan
> 
> 2011/8/2 Tim Burt <tcburt@rochester.rr.com>:
> >
> > I want to gather data from properties into something that can be used by
> > a babel source block (e.g. plot the data).  Searches in the manual,
> > worg, and gmane have not yielded the method, but my best guess is that
> > I've missed it.  If so, this is simply a request for a pointer to the
> > documentation I should read.
> >
> > BACKGROUND: I'm trying to complete a workflow where I capture data over
> > time and periodically operate with it (e.g. put into a database and
> > plot).  The excellent documentation for orgmode has made certain parts
> > easy going, but the current obstacle is extracting properties that can
> > subsequently be operated on in babel source blocks.
> >  + [X] Capture data into properties using templates
> >  + [X] View tables of properties in org file
> >   * [X] use columnview dynamic block
> >   * [X] use org-collector
> >  + [ ] Gather properties for use by babel source block
> >  + [-] Operate on the gathered properties
> >   * [-] SQL or SQLite
> >     - [X] follow simple tutorials for proof of principle
> >     - [ ] populate database with gathered properties
> >   * [-] R
> >     - [X] follow simple tutorials
> >     - [ ] calculate statistics and populate database
> >     - [ ] plot from database
> >     - [ ] plot from table in org file
> >
> > Good evening,
> > Tim
> >
> >

Re: Gather properties for use by babel source block?

[suvayu ali]

Hello Tim,

On Wed, Aug 3, 2011 at 4:35 PM,  <tcburt@rochester.rr.com> wrote:
> While I have the table from org-collector I do not know how to "easily pass it to code blocks".

I have put an example of passing tables to gnuplot source blocks below.
I hope you can adapt it to the source block of your choice. It also
illustrates the noweb syntax.

I hope this helps.

-------8<----------------8<-------------

#+BABEL: :noweb yes :results silent

* Gnuplot source preamble                                               :src:
  #+srcname: gnuplot-preamble
  #+begin_src gnuplot
    reset
    set terminal pdfcairo color size 21cm,14.8cm
    set termoption enhanced
    set termoption font "DejaVuSerif,8"
    set grid back
    set style line 1 linewidth 2 linecolor rgb 'forest-green'
    set style line 2 linewidth 2 linecolor rgb 'red'
    set label '(pp \@ 7 TeV)' at graph -0.08,1.05
    set ylabel 'K-factor'
    set xlabel 'W invariant mass (GeV)'
    set title 'K-factor = {/Symbol s}_{NLO}/{/Symbol s}_{LO}'
  #+end_src

* HORACE K-factor
** Gnuplot source							:src:
#+srcname: plot-horace-k-factor
#+begin_src gnuplot :var data=horace-kfactor-nnlo-vs-nnlo-w-scale-err
  <<gnuplot-preamble>>
  set output "horace-kfactor.pdf"
  set label 'Standard Model W' at graph 0.5,0.67
  set xrange [0:1000]
  plot "$data" using 1:2 with linespoints linestyle 1 title 'Horace K-factor'
  set output
#+end_src

** K-factor with uncertainties

#+TBLNAME: horace-kfactor-nnlo-vs-nnlo-w-scale-err
| M (GeV) | K-factor |
|---------+----------|
|      50 |    1.075 |
|     150 |     1.01 |
|     250 |   0.9932 |
|     350 |   0.9778 |
|     450 |   0.9621 |
|     550 |   0.9498 |
|     650 |   0.9365 |
|     750 |   0.9253 |
|     850 |   0.9122 |
|     950 |   0.9001 |

-------8<----------------8<-------------

-- 
Suvayu

Open source is the future. It sets us free.

Re: Gather properties for use by babel source block?

[Colin Hall]

Tim Burt <tcburt <at> rochester.rr.com> writes:
>
> I want to gather data from properties into something that can be used by
> a babel source block (e.g. plot the data).  Searches in the manual,
> worg, and gmane have not yielded the method, but my best guess is that
> I've missed it.  If so, this is simply a request for a pointer to the
> documentation I should read.

Any luck with this, Tim? I'm trying to do something very similar.

Cheers,
Colin.

Re: Gather properties for use by babel source block?

[Eric Schulte]

Colin Hall <colinghall@gmail.com> writes:

> Tim Burt <tcburt <at> rochester.rr.com> writes:
>>
>> I want to gather data from properties into something that can be used by
>> a babel source block (e.g. plot the data).  Searches in the manual,
>> worg, and gmane have not yielded the method, but my best guess is that
>> I've missed it.  If so, this is simply a request for a pointer to the
>> documentation I should read.
>
> Any luck with this, Tim? I'm trying to do something very similar.
>

One approach would be to use the org-collector [1] from contrib/ to
collect properties into a table.  That table could then be fed as the
argument to a source code block.

Hope this helps,

Footnotes: 
[1]  http://orgmode.org/worg/org-contrib/org-collector.html

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: Gather properties for use by babel source block?

[Colin Hall]

On Mon, May 07, 2012 at 10:16:36AM -0400, Eric Schulte wrote:
> Colin Hall <colinghall@gmail.com> writes:
> 
> > Tim Burt <tcburt <at> rochester.rr.com> writes:
> >>
> >> I want to gather data from properties into something that can be used by
> >> a babel source block (e.g. plot the data).  Searches in the manual,
> >> worg, and gmane have not yielded the method, but my best guess is that
> >> I've missed it.  If so, this is simply a request for a pointer to the
> >> documentation I should read.
> >
> > Any luck with this, Tim? I'm trying to do something very similar.
> >
> 
> One approach would be to use the org-collector [1] from contrib/ to
> collect properties into a table.  That table could then be fed as the
> argument to a source code block.
> 
> Hope this helps,

Yes, thank you, that was very helpful indeed.

Cheers,
Colin.

-- 

Colin Hall

Re: Gather properties for use by babel source block?

[Tim Burt]

[-- Attachment #1: Type: text/plain, Size: 4335 bytes --]

Colin Hall <colinghall@gmail.com> writes:

> Tim Burt <tcburt <at> rochester.rr.com> writes:
>>
>> I want to gather data from properties into something that can be used by
>> a babel source block (e.g. plot the data).  Searches in the manual,
>> worg, and gmane have not yielded the method, but my best guess is that
>> I've missed it.  If so, this is simply a request for a pointer to the
>> documentation I should read.
>
> Any luck with this, Tim? I'm trying to do something very similar.

After the hints of Darlan and Suvayu last August I cobbled a workflow
described below.  Errors and instability in the early weeks have been
resolved into the current set of workable blemishes described below the
workflow.  The attachment is an org file that should be a working example
if gnuplot is installed and configured for org-babel.

Thank you Colin for asking the question about progress, because I should
have shared it with the mailing list long ago warts and all.



This is an example file for gathering and plotting health data (some of
which has been fudged to protect the guilty ;-).  The workflow is this:
 1. DAILY: Use capture templates to query for data into a datetree under
    the heading '2011' which has an ID property.  Each data headline is
    marked with tags for the data class (e.g. weight, blood sugar) that is
    used as a hook for collection.
 2. ONE-TIME: Define the plots, in this case using gnuplot (thanks to
    Suvayu for the noweb example), under the heading 'Calculation and
    Visualization'.  Another one-time setup is to define propview blocks that
    org-collector will populate for each data class (e.g. weight).  Both
    the ID and tags are used to extract the information.
 3. ONE-TIME: Another one-time setup is to define propview blocks that
    org-collector will populate for each data class (e.g. weight).  Both
    the ID and tags are used to extract the information and this is under
    the 'Summaries' heading.
    - A #+tblname is placed after the #+BEGIN to provide a hook for the
      babel blocks for plotting.
 4. ON-DEMAND: Refresh each collector block (in this case 4 separate ones),
    then org-babel-execute-buffer to generate the plots.

This method has been mostly working but it has a few weaknesses, borne mostly from
my ignorance. I haven't thought about the problem in several months, so
maybe the act of writing the questions will spur me to the act of making it
better.  Here are my observations and questions: 
 - At various times in the past year the ON-DEMAND step has not worked
   consistently, especially in regards to the #+tblname within the
   collector block.  Sometimes the refresh would fail, often by wiping out
   the existing static view and replacing with an empty line under
   the #+tblname.  No matter what happened in the past, it is stable enough
   for this post.
   - The problem could have been one of configuration (e.g. org-collector
     not loaded) or data (e.g. error in the date tree) or something else
     entirely.  I have not done sufficiently thorough troubleshooting to
     really identify the different problems.
 - Collection is done by matching the tags in the data headlines, but the
   same type of information can be gleaned from the properties themselves.
   How can a match invocation be crafted to use the properties?
   - This weakness comes straight from my ignorance of lisp.
 - Collection is done for each separate table which does not scale well as
   the number of tables increases.  Scaling itself is not the issue with
   just 4 tables, but forgetting to update each table is a slight problem.
   - Is there a way to automate the org-collector step so that one action
     updates all tables?
   - Of course, for the current story another solution would be to create
     one single data table with org-collector then modify the plotting
     routines to extract only the necessary columns.  I'd like the ability
     to do monolithic or separate then decide which to use for which
     problems. 
 - The capture templates yield blank properties that simply clutter the
   drawer and make data extraction logic slightly more complicated since
   existence alone is not a complete test.
   - Can a capture template be written to ask all the questions, but not
     yield a property if the answer is 'null'?



[-- Attachment #2: TCBHealth-Example.org --]
[-- Type: application/vnd.lotus-organizer, Size: 21674 bytes --]

[-- Attachment #3: Type: text/plain, Size: 147 bytes --]


>
> Cheers,
> Colin.
>
>
>
>
>
>

-- 
Tim Burt
www.rketburt.org
"It is healthful to every sane man to utter the art within him;" -- GK Chesterton

Re: Gather properties for use by babel source block?

[Colin Hall]

On Tue, May 08, 2012 at 06:42:00AM -0400, Tim Burt wrote:
> Colin Hall <colinghall@gmail.com> writes:
> 
> > Tim Burt <tcburt <at> rochester.rr.com> writes:
> >>
> >> I want to gather data from properties into something that can be used by
> >> a babel source block (e.g. plot the data).  Searches in the manual,
> >> worg, and gmane have not yielded the method, but my best guess is that
> >> I've missed it.  If so, this is simply a request for a pointer to the
> >> documentation I should read.
> >
> > Any luck with this, Tim? I'm trying to do something very similar.
> 
> After the hints of Darlan and Suvayu last August I cobbled a workflow
> described below.  Errors and instability in the early weeks have been
> resolved into the current set of workable blemishes described below the
> workflow.  The attachment is an org file that should be a working example
> if gnuplot is installed and configured for org-babel.
> 
> Thank you Colin for asking the question about progress, because I should
> have shared it with the mailing list long ago warts and all.

Thanks for the sample code, Tim, that's got me started.

Cheers,
Colin.

-- 

Colin Hall