From mboxrd@z Thu Jan 1 00:00:00 1970 From: Brett Viren Subject: Re: asynchronous code evaluation Date: Wed, 04 Sep 2013 15:20:26 -0400 Message-ID: References: <20130904080040.GE679@kuru.dyndns-at-home.com> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha1; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:55337) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VHIdX-0004Qu-HV for emacs-orgmode@gnu.org; Wed, 04 Sep 2013 15:21:13 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1VHIdR-0005J7-MW for emacs-orgmode@gnu.org; Wed, 04 Sep 2013 15:21:07 -0400 Received: from smtpgw.bnl.gov ([2620:10a:0:3::30]:51556 helo=iron4.sec.bnl.local) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VHIdR-0005FV-H6 for emacs-orgmode@gnu.org; Wed, 04 Sep 2013 15:21:01 -0400 In-Reply-To: (Johannes Rainer's message of "Wed, 4 Sep 2013 15:23:03 +0200") List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: Johannes Rainer Cc: emacs-orgmode@gnu.org --=-=-= Content-Type: text/plain Hi Johannes, Johannes Rainer writes: > well, I'm using emacs/org for my data analyses in R. I thus combine > documentation (i.e. the conclusions drawn from e.g. plots created in > R) and the R code to perform the analysis in my org file. Since I'm > analysing high throughput data some tasks to handle the data are quite > time consuming. A variation on Allen's Makefile approach, I structure my (Python) code to be idempotent. The "real" code never (re)runs unless its output either doesn't exist or is older than the input. Normally when org runs the code's higher level interface it's essentially a no-op and very fast. If something does change it's usually because I'm doing development and will be running the code outside of org enough that it's reached its "no-op state" before I go back into emacs to refresh/edit the org document. Also, I keep things synchronous to avoid having to multiple, competing processes running at once. What I want is to take this approach and generalize into a more formal workflow system. One which handles caching files and in-memory objects in a versioned store to provide data provenance and idempotent running in a general sense. There's already been some work on this ("bein", "pyutilib workflow" and "sumatra" to name some Python oriented ones) but so far nothing that "just fits". I'm in the middle of trying my hand at something better, but it's not yet usable. -Brett. --=-=-= Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iEYEARECAAYFAlInh/oACgkQEixH2Z0dKCxj4wCfUDdByTEYYZIrpTWVpNrQr37M qlgAoKBD8+TsGkcRxUOClJaq8UuhT5u2 =TDCG -----END PGP SIGNATURE----- --=-=-=--