Re: babel and long-running computations

[John Kitchin <jkitchin@andrew.cmu.edu>]

[-- Attachment #1: Type: text/plain, Size: 2014 bytes --]

You are describing what I see as a need to run processes asynchronously.

We use the Torque queue system for this. It takes care of scheduling and
running jobs on a cluster.

On Thu, Apr 17, 2014 at 6:52 AM, Christoph Groth
<christoph@grothesque.org>wrote:

> Hi Julian,
>
> > Generally, org-babel is suited for long running computations.  Its
> > caching can help you avoid rerunning code chunks.  And long runtime does
> > not conflict with the idea of reproducible research, it just may be not
> > very comfortable for the user.
>
> I agree of course that it’s generally a good idea to structure the
> analysis in small logical steps and to save intermediate results.  Let
> me narrow down my question then: does org-babel support working with
> tasks that take several minutes to execute?  By this I mean (most
> important first):
>
> - Not freezing the editor during the execution of a task
>

You code-block should exit if a job is submitted, and record the job-id so
that you can check it later. Alternatively, you could do this in ipython
notebooks if you have python functions that return jobids.


> - Being able to execute multiple independent tasks in parallel
>

No problem. this is what queue systems were designed for.


> - Being able to interrupt a running task
>

you do this with the queue commands, e.g. qdel jobid


> - Being able to inspect the incomplete output of a running task
>

This is just checking the output files in the running jobs directories.


>
> I’d love to hear about any frameworks or workflows that fulfill these
> requirements.
>
> Cheers
>
> Christoph
>
>
> PS.
>
> I’m using ipython notebooks but I’m not happy with them because of the
> freezing problem and the complete lack of isolation of tasks within a
> single notebook (they live in a common mutable namespace).  I think a
> useful framework must be “functional” at the highest level for caching
> and dependencies to be useful.
>
>
>

[-- Attachment #2: Type: text/html, Size: 3092 bytes --]