From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Charles C. Berry" Subject: Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel Date: Thu, 18 Jun 2015 13:20:18 -0700 Message-ID: References: Mime-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:43902) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z5gIt-0001KV-KW for emacs-orgmode@gnu.org; Thu, 18 Jun 2015 16:20:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Z5gIp-0007OF-6k for emacs-orgmode@gnu.org; Thu, 18 Jun 2015 16:20:51 -0400 Received: from iport-acv1-out.ucsd.edu ([132.239.0.176]:51293) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z5gIo-0007KZ-JR for emacs-orgmode@gnu.org; Thu, 18 Jun 2015 16:20:46 -0400 In-Reply-To: List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: Rainer M Krug Cc: Xebar Saram , William Denton , org mode On Thu, 18 Jun 2015, Rainer M Krug wrote: > "Charles C. Berry" writes: > >> On Wed, 17 Jun 2015, William Denton wrote: >> >>> On 17 June 2015, Xebar Saram wrote: >>> >>>> I do alot of modeling work that involves using huge datasets and run >>>> process intensive R processes (such as complex mixed models, Gamms etc). in >>>> R studio all works well yet when i use the orgmode eval on R code blocks it >>>> works well for small simple process but 90% of the time when dealing with >>>> complex models and bug data (up to 256GB) it will just freeze emacs/ess. >>>> sometimes i can C-c or C-g it and other times i need to physically kill >>>> emacs. >>> >>> I've been having the same problem for a while, but wasn't able to >>> isolate it any more than large data sets, lack of memory, and heavy >>> CPU usage. Sometimes everything hangs and I need to power cycle the >>> computer. :( >>> >> >> And you (both) have `ess-eval-visibly' set to nil, right? >> [snip: ox-ravel and how it might solve OP's problem] > > I have a similar workflow, only that I use a package like > approach, i.e. I tangle function definitions in a folder ./R, data into > ./data (which makes it possible to share org defined variables with R > running outside org) and scripts, i.e. the things which do a analysis, > import data, ... i.e. which might take long, into a folder ./scripts/. I > then add the usual R package infrastructure files (DESCRIPTION, > NAMESPACE, ...). > Then I have one file tangled into ./scripts/init.R: [snip: how and why to structure an analysis as an R package] > I am using this approach at the moment for a paper and which will also > result in a paper. By executing all the scripts, one will be able to do > import the raw data, do the analysis and create all graphs used in the > paper. > And by writing the paper in the form of a vignette that Sweave or knitr can render you have an R package that when installed processes the data and reproduces the paper in pdf format. ox-ravel will produce that vignette from Org mode. For example, the bioConductor package geneRxCluster [1] comes from an Org mode document that contains the C and R code as src blocks and a subtree with the vignette (Using geneRxCluster) that analyzes data, produces graphics, etc. tangle-ing the src blocks and exporting the vignette creates the package. HTH, Chuck [1] http://www.bioconductor.org/packages/geneRxCluster