Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel

Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel

[Xebar Saram]

[-- Attachment #1: Type: text/plain, Size: 1633 bytes --]

Hi all

im not sure if this is absolutely the correct forum to raise this and if
not would be also happy to get input on where to persue this issue.

i recently dived into using orgmode, ESS, Babel etc to run code and im
really love it. The problem i have (and im not sure if its a org,Emacs or
ess issue) is that emacs sometimes (and mostly when dealing with R
processes involving HUGE databases) will just hang/freeze.

I do alot of modeling work that involves using huge datasets and run
process intensive R processes (such as complex mixed models, Gamms etc). in
R studio all works well yet when i use the orgmode eval on R code blocks it
works well for small simple process but 90% of the time when dealing with
complex models and bug data (up to 256GB) it will just freeze emacs/ess.
sometimes i can C-c or C-g it and other times i need to physically kill
emacs.

here is an example of such process that hangs


**** lmer
run the lmer part regressing stage 2 pred Vs mean pm

#+BEGIN_SRC R  :session Rorg  :results none
m2.smooth = lme(pred.m2 ~ meanPM25,random = list(aodid= ~1 +
meanPM25),control=lmeControl(opt = "optim"), data= mod2 )
#correlate to see everything from mod2 and the mpm works
mod2[, pred.t31 := predict(m2.smooth)]
mod2[, resid  := residuals(m2.smooth)]
print(summary(lm(pred.m2~pred.t31,data=mod2))$r.squared)
#+END_SRC

i usually issue org-babel-execute-subtree to eval several subsections under
a main header.

again i dont know if its an org mode isse perse but would love to hear from
people that have experience using R/org with big data/RAM and maybe point
me to where to raise these issues

best

Z

[-- Attachment #2: Type: text/html, Size: 2005 bytes --]

Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel

[William Denton]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 811 bytes --]

On 17 June 2015, Xebar Saram wrote:

> I do alot of modeling work that involves using huge datasets and run
> process intensive R processes (such as complex mixed models, Gamms etc). in
> R studio all works well yet when i use the orgmode eval on R code blocks it
> works well for small simple process but 90% of the time when dealing with
> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
> sometimes i can C-c or C-g it and other times i need to physically kill
> emacs.

I've been having the same problem for a while, but wasn't able to isolate it any 
more than large data sets, lack of memory, and heavy CPU usage.  Sometimes 
everything hangs and I need to power cycle the computer. :(

Bill
-- 
William Denton ↔  Toronto, Canada ↔  https://www.miskatonic.org/

Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel

[Charles C. Berry]

On Wed, 17 Jun 2015, William Denton wrote:

> On 17 June 2015, Xebar Saram wrote:
>
>> I do alot of modeling work that involves using huge datasets and run
>> process intensive R processes (such as complex mixed models, Gamms etc). in
>> R studio all works well yet when i use the orgmode eval on R code blocks it
>> works well for small simple process but 90% of the time when dealing with
>> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
>> sometimes i can C-c or C-g it and other times i need to physically kill
>> emacs.
>
> I've been having the same problem for a while, but wasn't able to isolate it 
> any more than large data sets, lack of memory, and heavy CPU usage. 
> Sometimes everything hangs and I need to power cycle the computer. :(
>

And you (both) have `ess-eval-visibly' set to nil, right?

I do statistical genomics, which can be compute intensive. Sometimes 
processes need to run for a while, and I get impatient having to wait.

I wrote (and use) ox-ravel[1] to speed up my write-run-revise cycle in 
org-mode.

Basically, ravel will export Org mode to a format that knitr (and the 
like) can run - turning src blocks into `code chunks'. That allows me to 
set the cache=TRUE chunk option, etc. I run knitr on the exported document 
to initialize objects for long running computations or to produce a 
finished report.

When I start a session, I run knitr in the R session, then all the cached 
objects are loaded in and ready to use.

If I write a src block I know will take a long time to export, I export 
from org mode to update the knitr document and re-knit it to refresh the 
cache.

Mostly, I work in org-mode adding src blocks, revising existing ones, or 
editing text and graphics.

If you decide to try ravel I recommend the `ravel-lang' branch[2] as that 
will soon replace master.

HTH,

Chuck


[1] https://github.com/chasberry/orgmode-accessories
[2] https://github.com/chasberry/orgmode-accessories/tree/ravel-lang

Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel

[Xebar Saram]

[-- Attachment #1: Type: text/plain, Size: 2361 bytes --]

Thx Chuck

this sounds great. could you perhaps point us to some documentation on
this, or perhpas consider sharing a detailed overview of your workflow?
this seems it could really fit my needs.

thx so much in advance

Z


On Thu, Jun 18, 2015 at 6:17 AM, Charles C. Berry <ccberry@ucsd.edu> wrote:

> On Wed, 17 Jun 2015, William Denton wrote:
>
>  On 17 June 2015, Xebar Saram wrote:
>>
>>  I do alot of modeling work that involves using huge datasets and run
>>> process intensive R processes (such as complex mixed models, Gamms etc).
>>> in
>>> R studio all works well yet when i use the orgmode eval on R code blocks
>>> it
>>> works well for small simple process but 90% of the time when dealing with
>>> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
>>> sometimes i can C-c or C-g it and other times i need to physically kill
>>> emacs.
>>>
>>
>> I've been having the same problem for a while, but wasn't able to isolate
>> it any more than large data sets, lack of memory, and heavy CPU usage.
>> Sometimes everything hangs and I need to power cycle the computer. :(
>>
>>
> And you (both) have `ess-eval-visibly' set to nil, right?
>
> I do statistical genomics, which can be compute intensive. Sometimes
> processes need to run for a while, and I get impatient having to wait.
>
> I wrote (and use) ox-ravel[1] to speed up my write-run-revise cycle in
> org-mode.
>
> Basically, ravel will export Org mode to a format that knitr (and the
> like) can run - turning src blocks into `code chunks'. That allows me to
> set the cache=TRUE chunk option, etc. I run knitr on the exported document
> to initialize objects for long running computations or to produce a
> finished report.
>
> When I start a session, I run knitr in the R session, then all the cached
> objects are loaded in and ready to use.
>
> If I write a src block I know will take a long time to export, I export
> from org mode to update the knitr document and re-knit it to refresh the
> cache.
>
> Mostly, I work in org-mode adding src blocks, revising existing ones, or
> editing text and graphics.
>
> If you decide to try ravel I recommend the `ravel-lang' branch[2] as that
> will soon replace master.
>
> HTH,
>
> Chuck
>
>
> [1] https://github.com/chasberry/orgmode-accessories
> [2] https://github.com/chasberry/orgmode-accessories/tree/ravel-lang
>

[-- Attachment #2: Type: text/html, Size: 3320 bytes --]

Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel

[Charles C. Berry]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1906 bytes --]

On Thu, 18 Jun 2015, Xebar Saram wrote:

> Thx Chuck
>
> this sounds great. could you perhaps point us to some documentation on
> this, or perhpas consider sharing a detailed overview of your workflow?
> this seems it could really fit my needs.

Start with Sections 1-3 of ox-ravel.org for basic info.

There are a couple of examples on my github repo. Looking at the 
ravel-lang branch[1], there are these files:

- example-1-Rnw.org :: contains an org mode translation of the
      “example-1.Rnw’ (Sweave) file from the R distribution and
      instructions on how to export it.


- knitr-minimal-rhtml.org :: contains the “knitr-minimal.Rhtml’ file
      from the knitr demos page, but modified to *.org format.


- demos.org :: contains a variety of examples. It is best viewed
      online using 'raw' (or by downloading and viewing it in Org mode),
      as github formatting masks the #+BEGIN_EXAMPLE ... #+END_EXAMPLE
      sections that show what the output should be.

As far as caching goes this file (named cache.org, say)

--8<---------------cut here---------------start------------->8---

* cache this chunk

#+ATTR_RAVEL: cache=TRUE
#+NAME: show-time
#+BEGIN_SRC R
   firstTime <- date()
   firstTime
#+END_SRC

--8<---------------cut here---------------end--------------->8---

when exported as ravel-latex (i.e. C-c C-e r l, if you have ox-ravel
up and running, see section 2 of ox-ravel.org) creates cache.Rnw with
one chunk for which caching is specified. Loading knitr and running
knit("cache.Rnw") in R produces cache.tex and 3 ./cache/show-time*
files. Rerunning cache.Rnw will not update those files even if
cache.Rnw is updated as long as the code in the show-time block is
unchanged. You can start a fresh session, run knit("cache.Rnw"), and
the value of firstTime will be loaded into it.

HTH,

Chuck

[1] https://github.com/chasberry/orgmode-accessories/tree/ravel-lang

Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel

[Rainer M Krug]

[-- Attachment #1: Type: text/plain, Size: 4898 bytes --]

"Charles C. Berry" <ccberry@ucsd.edu> writes:

> On Wed, 17 Jun 2015, William Denton wrote:
>
>> On 17 June 2015, Xebar Saram wrote:
>>
>>> I do alot of modeling work that involves using huge datasets and run
>>> process intensive R processes (such as complex mixed models, Gamms etc). in
>>> R studio all works well yet when i use the orgmode eval on R code blocks it
>>> works well for small simple process but 90% of the time when dealing with
>>> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
>>> sometimes i can C-c or C-g it and other times i need to physically kill
>>> emacs.
>>
>> I've been having the same problem for a while, but wasn't able to
>> isolate it any more than large data sets, lack of memory, and heavy
>> CPU usage. Sometimes everything hangs and I need to power cycle the
>> computer. :(
>>
>
> And you (both) have `ess-eval-visibly' set to nil, right?
>
> I do statistical genomics, which can be compute intensive. Sometimes
> processes need to run for a while, and I get impatient having to wait.
>
> I wrote (and use) ox-ravel[1] to speed up my write-run-revise cycle in
> org-mode.
>
> Basically, ravel will export Org mode to a format that knitr (and the
> like) can run - turning src blocks into `code chunks'. That allows me
> to set the cache=TRUE chunk option, etc. I run knitr on the exported
> document to initialize objects for long running computations or to
> produce a finished report.
>
> When I start a session, I run knitr in the R session, then all the
> cached objects are loaded in and ready to use.
>
> If I write a src block I know will take a long time to export, I
> export from org mode to update the knitr document and re-knit it to
> refresh the cache.

I have a similar workflow, only that I use a package like
approach, i.e. I tangle function definitions in a folder ./R, data into
./data (which makes it possible to share org defined variables with R
running outside org) and scripts, i.e. the things which do a analysis,
import data, ... i.e. which might take long, into a folder ./scripts/. I
then add the usual R package infrastructure files (DESCRIPTION,
NAMESPACE, ...).
Then I have one file tangled into ./scripts/init.R:

#+begin_src R :tangle ./scripts/init.R  
library(devtools)
load_all()
#+end_src

and one for the analysis:

#+begin_src R :tangle ./scripts/myAnalysis.R  
## Do some really time intensive and horribly complicated and important
## stuff here
save(
    fileNames,
    bw,
    cols,
    labels,
    fit,
    dens,
    gof,
    gofPerProf,
    file = "./cache/results.myAnalysis.rds"
)
#+end_src

Now after tangling, I have my code easily available in a new R session:

1) start R in the directory in which the DESCRIPTION file is, 
2) run source("./scripts/init.R")

and I have all my functions and data available.

To run a analysis, I do

3) source("./scripts/myAnalysis.R")

and the results are saved in a file fn

To analyse the data further, I can then simply use

#+begin_src R :tangle ./scripts/myAnalysis.R
fitSing <- attach("./cache/results.myAnalysis.rds")
#+end_src

so they won't interfere with my environment in R.

I can finally remove the attached environment by doing

#+begin_src R :tangle ./scripts/myAnalysis.R  
detach(
    name = attr(fitSing, "name"),
    character.only = TRUE
)
#+end_src

Through these caching and compartmentalizing, I can easily do some
things outside org and some inside, and easily combine all the data.

Further advantage: I can actually create the package and send it to
somebody for testing and review and it should run out of the box, as in
the DESCRIPTION file all dependencies are defined.

I am using this approach at the moment for a paper and which will also
result in a paper. By executing all the scripts, one will be able to do
import the raw data, do the analysis and create all graphs used in the
paper.

Hope this gives you another idea how one can handle long running
analysis in R in org,

Cheers,

Rainer

>
> Mostly, I work in org-mode adding src blocks, revising existing ones,
> or editing text and graphics.
>
> If you decide to try ravel I recommend the `ravel-lang' branch[2] as
> that will soon replace master.
>
> HTH,
>
> Chuck
>
>
> [1] https://github.com/chasberry/orgmode-accessories
> [2] https://github.com/chasberry/orgmode-accessories/tree/ravel-lang
>

-- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :       +33 - (0)9 53 10 27 44
Cell:       +33 - (0)6 85 62 59 98
Fax :       +33 - (0)9 58 10 27 44

Fax (D):    +49 - (0)3 21 21 25 22 44

email:      Rainer@krugs.de

Skype:      RMkrug

PGP: 0x0F52F982

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 480 bytes --]

Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel

[Charles C. Berry]

On Thu, 18 Jun 2015, Rainer M Krug wrote:

> "Charles C. Berry" <ccberry@ucsd.edu> writes:
>
>> On Wed, 17 Jun 2015, William Denton wrote:
>>
>>> On 17 June 2015, Xebar Saram wrote:
>>>
>>>> I do alot of modeling work that involves using huge datasets and run
>>>> process intensive R processes (such as complex mixed models, Gamms etc). in
>>>> R studio all works well yet when i use the orgmode eval on R code blocks it
>>>> works well for small simple process but 90% of the time when dealing with
>>>> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
>>>> sometimes i can C-c or C-g it and other times i need to physically kill
>>>> emacs.
>>>
>>> I've been having the same problem for a while, but wasn't able to
>>> isolate it any more than large data sets, lack of memory, and heavy
>>> CPU usage. Sometimes everything hangs and I need to power cycle the
>>> computer. :(
>>>
>>
>> And you (both) have `ess-eval-visibly' set to nil, right?
>>

[snip: ox-ravel and how it might solve OP's problem]

>
> I have a similar workflow, only that I use a package like
> approach, i.e. I tangle function definitions in a folder ./R, data into
> ./data (which makes it possible to share org defined variables with R
> running outside org) and scripts, i.e. the things which do a analysis,
> import data, ... i.e. which might take long, into a folder ./scripts/. I
> then add the usual R package infrastructure files (DESCRIPTION,
> NAMESPACE, ...).
> Then I have one file tangled into ./scripts/init.R:

[snip: how and why to structure an analysis as an R package]


> I am using this approach at the moment for a paper and which will also
> result in a paper. By executing all the scripts, one will be able to do
> import the raw data, do the analysis and create all graphs used in the
> paper.
>

And by writing the paper in the form of a vignette that Sweave or knitr 
can render you have an R package that when installed processes the data 
and reproduces the paper in pdf format.

ox-ravel will produce that vignette from Org mode.

For example, the bioConductor package geneRxCluster [1] comes from an Org 
mode document that contains the C and R code as src blocks and a subtree 
with the vignette (Using geneRxCluster) that analyzes data, produces 
graphics, etc. tangle-ing the src blocks and exporting the vignette 
creates the package.

HTH,

Chuck

[1] http://www.bioconductor.org/packages/geneRxCluster

Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel

[Andreas Leha]

Hi Chuck,

[snip: all context about workflows for R projects]

> For example, the bioConductor package geneRxCluster [1] comes from an
> Org mode document that contains the C and R code as src blocks and a
> subtree with the vignette (Using geneRxCluster) that analyzes data,
> produces graphics, etc. tangle-ing the src blocks and exporting the
> vignette creates the package.
>

I'd love to see that org document, but I do not seem to be able to find
it.  Is it available anywhere?

Thanks,
Andreas

Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel

[Charles C. Berry]

On Thu, 18 Jun 2015, Andreas Leha wrote:

> Hi Chuck,
>
> [snip: all context about workflows for R projects]
>
>> For example, the bioConductor package geneRxCluster [1] comes from an
>> Org mode document that contains the C and R code as src blocks and a
>> subtree with the vignette (Using geneRxCluster) that analyzes data,
>> produces graphics, etc. tangle-ing the src blocks and exporting the
>> vignette creates the package.
>>
>
> I'd love to see that org document, but I do not seem to be able to find
> it.  Is it available anywhere?

Andreas,

It is now, or at least the relevant subset of it is at:

https://github.com/chasberry/geneRx/blob/master/Rpackage.org

It is best viewed in Org mode as there are internal links that github does 
not honor (so you get `404 This is not...' messages if you click on them).

And it is best to use those internal links as the file was not designed to 
be read from top to bottom - it has lots of snippets I used to develop and 
check code along the way that are not in the actual package and I have 
left them in place.

Best,

Chuck

Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel

[Andreas Leha]

"Charles C. Berry" <ccberry@ucsd.edu> writes:
> On Thu, 18 Jun 2015, Andreas Leha wrote:
>
>> Hi Chuck,
>>
>> [snip: all context about workflows for R projects]
>>
>>> For example, the bioConductor package geneRxCluster [1] comes from an
>>> Org mode document that contains the C and R code as src blocks and a
>>> subtree with the vignette (Using geneRxCluster) that analyzes data,
>>> produces graphics, etc. tangle-ing the src blocks and exporting the
>>> vignette creates the package.
>>>
>>
>> I'd love to see that org document, but I do not seem to be able to find
>> it.  Is it available anywhere?
>
> Andreas,
>
> It is now, or at least the relevant subset of it is at:
>
> https://github.com/chasberry/geneRx/blob/master/Rpackage.org
>
> It is best viewed in Org mode as there are internal links that github
> does not honor (so you get `404 This is not...' messages if you click
> on them).
>
> And it is best to use those internal links as the file was not
> designed to be read from top to bottom - it has lots of snippets I
> used to develop and check code along the way that are not in the
> actual package and I have left them in place.
>

That is awesome!  Really appreciated!  I will take a good look.

Thanks,
Andreas

Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel

[Andreas Leha]

Hi Rainer,

Rainer M Krug <Rainer@krugs.de> writes:
> "Charles C. Berry" <ccberry@ucsd.edu> writes:
>
>> On Wed, 17 Jun 2015, William Denton wrote:
>>
>>> On 17 June 2015, Xebar Saram wrote:
>>>
>>>> I do alot of modeling work that involves using huge datasets and run
>>>> process intensive R processes (such as complex mixed models, Gamms etc). in
>>>> R studio all works well yet when i use the orgmode eval on R code blocks it
>>>> works well for small simple process but 90% of the time when dealing with
>>>> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
>>>> sometimes i can C-c or C-g it and other times i need to physically kill
>>>> emacs.
>>>
>>> I've been having the same problem for a while, but wasn't able to
>>> isolate it any more than large data sets, lack of memory, and heavy
>>> CPU usage. Sometimes everything hangs and I need to power cycle the
>>> computer. :(
>>>
>>
>> And you (both) have `ess-eval-visibly' set to nil, right?
>>
>> I do statistical genomics, which can be compute intensive. Sometimes
>> processes need to run for a while, and I get impatient having to wait.
>>
>> I wrote (and use) ox-ravel[1] to speed up my write-run-revise cycle in
>> org-mode.
>>
>> Basically, ravel will export Org mode to a format that knitr (and the
>> like) can run - turning src blocks into `code chunks'. That allows me
>> to set the cache=TRUE chunk option, etc. I run knitr on the exported
>> document to initialize objects for long running computations or to
>> produce a finished report.
>>
>> When I start a session, I run knitr in the R session, then all the
>> cached objects are loaded in and ready to use.
>>
>> If I write a src block I know will take a long time to export, I
>> export from org mode to update the knitr document and re-knit it to
>> refresh the cache.
>
> I have a similar workflow, only that I use a package like
> approach, i.e. I tangle function definitions in a folder ./R, data into
> ./data (which makes it possible to share org defined variables with R
> running outside org) and scripts, i.e. the things which do a analysis,
> import data, ... i.e. which might take long, into a folder ./scripts/. I
> then add the usual R package infrastructure files (DESCRIPTION,
> NAMESPACE, ...).
> Then I have one file tangled into ./scripts/init.R:
>
> #+begin_src R :tangle ./scripts/init.R  
> library(devtools)
> load_all()
> #+end_src
>
>
> and one for the analysis:
>
> #+begin_src R :tangle ./scripts/myAnalysis.R  
> ## Do some really time intensive and horribly complicated and important
> ## stuff here
> save(
>     fileNames,
>     bw,
>     cols,
>     labels,
>     fit,
>     dens,
>     gof,
>     gofPerProf,
>     file = "./cache/results.myAnalysis.rds"
> )
> #+end_src
>
>
> Now after tangling, I have my code easily available in a new R session:
>
> 1) start R in the directory in which the DESCRIPTION file is, 
> 2) run source("./scripts/init.R")
>
> and I have all my functions and data available.
>
> To run a analysis, I do
>
> 3) source("./scripts/myAnalysis.R")
>
> and the results are saved in a file fn
>
> To analyse the data further, I can then simply use
>
> #+begin_src R :tangle ./scripts/myAnalysis.R
> fitSing <- attach("./cache/results.myAnalysis.rds")
> #+end_src
>
>
> so they won't interfere with my environment in R.
>
> I can finally remove the attached environment by doing
>
> #+begin_src R :tangle ./scripts/myAnalysis.R  
> detach(
>     name = attr(fitSing, "name"),
>     character.only = TRUE
> )
> #+end_src
>
> Through these caching and compartmentalizing, I can easily do some
> things outside org and some inside, and easily combine all the data.
>
> Further advantage: I can actually create the package and send it to
> somebody for testing and review and it should run out of the box, as in
> the DESCRIPTION file all dependencies are defined.
>
> I am using this approach at the moment for a paper and which will also
> result in a paper. By executing all the scripts, one will be able to do
> import the raw data, do the analysis and create all graphs used in the
> paper.
>
> Hope this gives you another idea how one can handle long running
> analysis in R in org,
>
> Cheers,
>
> Rainer
>

That is a cool workflow.  I especially like the fact that you end up
with an R package.

So, I'll try my again.   Is there there any chance to see working
example of this?  I'd love to see that.

Thanks,
Andreas

Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel

[Rainer M Krug]

[-- Attachment #1: Type: text/plain, Size: 5638 bytes --]

Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:

> Hi Rainer,

Hi Andreas,

>
> Rainer M Krug <Rainer@krugs.de> writes:
>> "Charles C. Berry" <ccberry@ucsd.edu> writes:
>>
>>> On Wed, 17 Jun 2015, William Denton wrote:
>>>
>>>> On 17 June 2015, Xebar Saram wrote:
>>>>
>>>>> I do alot of modeling work that involves using huge datasets and run
>>>>> process intensive R processes (such as complex mixed models, Gamms etc). in
>>>>> R studio all works well yet when i use the orgmode eval on R code blocks it
>>>>> works well for small simple process but 90% of the time when dealing with
>>>>> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
>>>>> sometimes i can C-c or C-g it and other times i need to physically kill
>>>>> emacs.
>>>>
>>>> I've been having the same problem for a while, but wasn't able to
>>>> isolate it any more than large data sets, lack of memory, and heavy
>>>> CPU usage. Sometimes everything hangs and I need to power cycle the
>>>> computer. :(
>>>>
>>>
>>> And you (both) have `ess-eval-visibly' set to nil, right?
>>>
>>> I do statistical genomics, which can be compute intensive. Sometimes
>>> processes need to run for a while, and I get impatient having to wait.
>>>
>>> I wrote (and use) ox-ravel[1] to speed up my write-run-revise cycle in
>>> org-mode.
>>>
>>> Basically, ravel will export Org mode to a format that knitr (and the
>>> like) can run - turning src blocks into `code chunks'. That allows me
>>> to set the cache=TRUE chunk option, etc. I run knitr on the exported
>>> document to initialize objects for long running computations or to
>>> produce a finished report.
>>>
>>> When I start a session, I run knitr in the R session, then all the
>>> cached objects are loaded in and ready to use.
>>>
>>> If I write a src block I know will take a long time to export, I
>>> export from org mode to update the knitr document and re-knit it to
>>> refresh the cache.
>>
>> I have a similar workflow, only that I use a package like
>> approach, i.e. I tangle function definitions in a folder ./R, data into
>> ./data (which makes it possible to share org defined variables with R
>> running outside org) and scripts, i.e. the things which do a analysis,
>> import data, ... i.e. which might take long, into a folder ./scripts/. I
>> then add the usual R package infrastructure files (DESCRIPTION,
>> NAMESPACE, ...).
>> Then I have one file tangled into ./scripts/init.R:
>>
>> #+begin_src R :tangle ./scripts/init.R  
>> library(devtools)
>> load_all()
>> #+end_src
>>
>>
>> and one for the analysis:
>>
>> #+begin_src R :tangle ./scripts/myAnalysis.R  
>> ## Do some really time intensive and horribly complicated and important
>> ## stuff here
>> save(
>>     fileNames,
>>     bw,
>>     cols,
>>     labels,
>>     fit,
>>     dens,
>>     gof,
>>     gofPerProf,
>>     file = "./cache/results.myAnalysis.rds"
>> )
>> #+end_src
>>
>>
>> Now after tangling, I have my code easily available in a new R session:
>>
>> 1) start R in the directory in which the DESCRIPTION file is, 
>> 2) run source("./scripts/init.R")
>>
>> and I have all my functions and data available.
>>
>> To run a analysis, I do
>>
>> 3) source("./scripts/myAnalysis.R")
>>
>> and the results are saved in a file fn
>>
>> To analyse the data further, I can then simply use
>>
>> #+begin_src R :tangle ./scripts/myAnalysis.R
>> fitSing <- attach("./cache/results.myAnalysis.rds")
>> #+end_src
>>
>>
>> so they won't interfere with my environment in R.
>>
>> I can finally remove the attached environment by doing
>>
>> #+begin_src R :tangle ./scripts/myAnalysis.R  
>> detach(
>>     name = attr(fitSing, "name"),
>>     character.only = TRUE
>> )
>> #+end_src
>>
>> Through these caching and compartmentalizing, I can easily do some
>> things outside org and some inside, and easily combine all the data.
>>
>> Further advantage: I can actually create the package and send it to
>> somebody for testing and review and it should run out of the box, as in
>> the DESCRIPTION file all dependencies are defined.
>>
>> I am using this approach at the moment for a paper and which will also
>> result in a paper. By executing all the scripts, one will be able to do
>> import the raw data, do the analysis and create all graphs used in the
>> paper.
>>
>> Hope this gives you another idea how one can handle long running
>> analysis in R in org,
>>
>> Cheers,
>>
>> Rainer
>>
>
> That is a cool workflow.  I especially like the fact that you end up
> with an R package.

Thanks. Yes - the idea of having a package at the end was one main
reason why I am using this approach.


>
> So, I'll try my again.   Is there there any chance to see working
> example of this?  I'd love to see that.

Let's say I am working on it. I am working on a project which is using
this workflow and when it is finished, the package will be available as
an electronic appendix to the paper.

But I will see if I can condense an example and blog it - I'll let you
kow when it is done.

Cheers,

Rainer



>
> Thanks,
> Andreas
>
>

-- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :       +33 - (0)9 53 10 27 44
Cell:       +33 - (0)6 85 62 59 98
Fax :       +33 - (0)9 58 10 27 44

Fax (D):    +49 - (0)3 21 21 25 22 44

email:      Rainer@krugs.de

Skype:      RMkrug

PGP: 0x0F52F982

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 480 bytes --]

Re: Emacs/ESS/org freezes/hangs on big data/ RAM(~256GB) processes when run in org/babel

[Andreas Leha]

Rainer M Krug <Rainer@krugs.de> writes:
> Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:
>
>> Hi Rainer,
>
> Hi Andreas,
>
>>
>> Rainer M Krug <Rainer@krugs.de> writes:
>>> "Charles C. Berry" <ccberry@ucsd.edu> writes:
>>>
>>>> On Wed, 17 Jun 2015, William Denton wrote:
>>>>
>>>>> On 17 June 2015, Xebar Saram wrote:
>>>>>
>>>>>> I do alot of modeling work that involves using huge datasets and run
>>>>>> process intensive R processes (such as complex mixed models, Gamms etc). in
>>>>>> R studio all works well yet when i use the orgmode eval on R code blocks it
>>>>>> works well for small simple process but 90% of the time when dealing with
>>>>>> complex models and bug data (up to 256GB) it will just freeze emacs/ess.
>>>>>> sometimes i can C-c or C-g it and other times i need to physically kill
>>>>>> emacs.
>>>>>
>>>>> I've been having the same problem for a while, but wasn't able to
>>>>> isolate it any more than large data sets, lack of memory, and heavy
>>>>> CPU usage. Sometimes everything hangs and I need to power cycle the
>>>>> computer. :(
>>>>>
>>>>
>>>> And you (both) have `ess-eval-visibly' set to nil, right?
>>>>
>>>> I do statistical genomics, which can be compute intensive. Sometimes
>>>> processes need to run for a while, and I get impatient having to wait.
>>>>
>>>> I wrote (and use) ox-ravel[1] to speed up my write-run-revise cycle in
>>>> org-mode.
>>>>
>>>> Basically, ravel will export Org mode to a format that knitr (and the
>>>> like) can run - turning src blocks into `code chunks'. That allows me
>>>> to set the cache=TRUE chunk option, etc. I run knitr on the exported
>>>> document to initialize objects for long running computations or to
>>>> produce a finished report.
>>>>
>>>> When I start a session, I run knitr in the R session, then all the
>>>> cached objects are loaded in and ready to use.
>>>>
>>>> If I write a src block I know will take a long time to export, I
>>>> export from org mode to update the knitr document and re-knit it to
>>>> refresh the cache.
>>>
>>> I have a similar workflow, only that I use a package like
>>> approach, i.e. I tangle function definitions in a folder ./R, data into
>>> ./data (which makes it possible to share org defined variables with R
>>> running outside org) and scripts, i.e. the things which do a analysis,
>>> import data, ... i.e. which might take long, into a folder ./scripts/. I
>>> then add the usual R package infrastructure files (DESCRIPTION,
>>> NAMESPACE, ...).
>>> Then I have one file tangled into ./scripts/init.R:
>>>
>>> #+begin_src R :tangle ./scripts/init.R  
>>> library(devtools)
>>> load_all()
>>> #+end_src
>>>
>>>
>>> and one for the analysis:
>>>
>>> #+begin_src R :tangle ./scripts/myAnalysis.R  
>>> ## Do some really time intensive and horribly complicated and important
>>> ## stuff here
>>> save(
>>>     fileNames,
>>>     bw,
>>>     cols,
>>>     labels,
>>>     fit,
>>>     dens,
>>>     gof,
>>>     gofPerProf,
>>>     file = "./cache/results.myAnalysis.rds"
>>> )
>>> #+end_src
>>>
>>>
>>> Now after tangling, I have my code easily available in a new R session:
>>>
>>> 1) start R in the directory in which the DESCRIPTION file is, 
>>> 2) run source("./scripts/init.R")
>>>
>>> and I have all my functions and data available.
>>>
>>> To run a analysis, I do
>>>
>>> 3) source("./scripts/myAnalysis.R")
>>>
>>> and the results are saved in a file fn
>>>
>>> To analyse the data further, I can then simply use
>>>
>>> #+begin_src R :tangle ./scripts/myAnalysis.R
>>> fitSing <- attach("./cache/results.myAnalysis.rds")
>>> #+end_src
>>>
>>>
>>> so they won't interfere with my environment in R.
>>>
>>> I can finally remove the attached environment by doing
>>>
>>> #+begin_src R :tangle ./scripts/myAnalysis.R  
>>> detach(
>>>     name = attr(fitSing, "name"),
>>>     character.only = TRUE
>>> )
>>> #+end_src
>>>
>>> Through these caching and compartmentalizing, I can easily do some
>>> things outside org and some inside, and easily combine all the data.
>>>
>>> Further advantage: I can actually create the package and send it to
>>> somebody for testing and review and it should run out of the box, as in
>>> the DESCRIPTION file all dependencies are defined.
>>>
>>> I am using this approach at the moment for a paper and which will also
>>> result in a paper. By executing all the scripts, one will be able to do
>>> import the raw data, do the analysis and create all graphs used in the
>>> paper.
>>>
>>> Hope this gives you another idea how one can handle long running
>>> analysis in R in org,
>>>
>>> Cheers,
>>>
>>> Rainer
>>>
>>
>> That is a cool workflow.  I especially like the fact that you end up
>> with an R package.
>
> Thanks. Yes - the idea of having a package at the end was one main
> reason why I am using this approach.
>
>
>>
>> So, I'll try my again.   Is there there any chance to see working
>> example of this?  I'd love to see that.
>
> Let's say I am working on it. I am working on a project which is using
> this workflow and when it is finished, the package will be available as
> an electronic appendix to the paper.
>
> But I will see if I can condense an example and blog it - I'll let you
> kow when it is done.
>

Thanks!  Either way, I am really looking forward to this.

Regards,
Andreas