[PATCH] gnu: r: Update to 3.3.1.

[PATCH] gnu: r: Update to 3.3.1.

[Roel Janssen]

Dear Guix,

With the following patch I would like to update R to 3.3.1.  I tested
with R packages I regularly use and everything seems to work just fine.

Is it OK to upgrade?

Kind regards,
Roel Janssen


From e8cdef8b2f287d26c4a4a9a02a7277dc940ed667 Mon Sep 17 00:00:00 2001
From: Roel Janssen <roel@gnu.org>
Date: Sat, 30 Jul 2016 16:33:17 +0200
Subject: [PATCH] gnu: r: Update to 3.3.1.

* gnu/packages/statistics.scm (r): Update to 3.3.1.
---
 gnu/packages/statistics.scm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/statistics.scm b/gnu/packages/statistics.scm
index 2365149..ee84ffe 100644
--- a/gnu/packages/statistics.scm
+++ b/gnu/packages/statistics.scm
@@ -100,7 +100,7 @@ be output in text, PostScript, PDF or HTML.")
 (define-public r
   (package
     (name "r")
-    (version "3.3.0")
+    (version "3.3.1")
     (source (origin
               (method url-fetch)
               (uri (string-append "mirror://cran/src/base/R-"
@@ -108,7 +108,7 @@ be output in text, PostScript, PDF or HTML.")
                                   version ".tar.gz"))
               (sha256
                (base32
-                "1r0i0cqs3p0vrpiwq0zg5kbrmja9rmaijyzf9f23v6d5n5ab2mlj"))))
+                "1qm9znh8akfy9fkzzi6f1vz2w1dd0chsr6qn7kw80lqzhgjrmi9x"))))
     (build-system gnu-build-system)
     (arguments
      `(#:make-flags
-- 
2.9.2

Re: [PATCH] gnu: r: Update to 3.3.1.

[myglc2]

Roel Janssen <roel@gnu.org> writes:

> Dear Guix,
>
> With the following patch I would like to update R to 3.3.1.  I tested
> with R packages I regularly use and everything seems to work just fine.
>
> Is it OK to upgrade?
>
> Kind regards,
> Roel Janssen
>
>
>From e8cdef8b2f287d26c4a4a9a02a7277dc940ed667 Mon Sep 17 00:00:00 2001
> From: Roel Janssen <roel@gnu.org>
> Date: Sat, 30 Jul 2016 16:33:17 +0200
> Subject: [PATCH] gnu: r: Update to 3.3.1.
>
> * gnu/packages/statistics.scm (r): Update to 3.3.1.
> ---
>  gnu/packages/statistics.scm | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/gnu/packages/statistics.scm b/gnu/packages/statistics.scm
> index 2365149..ee84ffe 100644
> --- a/gnu/packages/statistics.scm
> +++ b/gnu/packages/statistics.scm
> @@ -100,7 +100,7 @@ be output in text, PostScript, PDF or HTML.")
>  (define-public r
>    (package
>      (name "r")
> -    (version "3.3.0")
> +    (version "3.3.1")
>      (source (origin
>                (method url-fetch)
>                (uri (string-append "mirror://cran/src/base/R-"
> @@ -108,7 +108,7 @@ be output in text, PostScript, PDF or HTML.")
>                                    version ".tar.gz"))
>                (sha256
>                 (base32
> -                "1r0i0cqs3p0vrpiwq0zg5kbrmja9rmaijyzf9f23v6d5n5ab2mlj"))))
> +                "1qm9znh8akfy9fkzzi6f1vz2w1dd0chsr6qn7kw80lqzhgjrmi9x"))))
>      (build-system gnu-build-system)
>      (arguments
>       `(#:make-flags

Hi Roel,

In my experience, the typical bio project analysis goes on for 1-5
years.  Meanwhile R and BioConductor releases take place. Unfortunately,
I have seen new R/BioConductor versions break a package in use, make it
unavailable, or modify an analysis result, thereby screwing up the
analysis at the most inopportune time ;-(

The workaround used by sysops where I work (hospital research lab) is to
give notice of R upgrades and to make previous releases available for
reference by ongoing projects. IMO, we should consider how the guix R
recipe(s) might support a pattern of use like this.

I can assure you that if our users do guix pull and invisibly get a new
R release, their analyses will from time to time break. So we may want a
simple way for them to back down to a previous release. So.. I am
thinking it would make sense to keep previous versions of R in the
recipe. What do others think?

- George

Re: [PATCH] gnu: r: Update to 3.3.1.

[Ludovic Courtès]

myglc2 <myglc2@gmail.com> skribis:

> I can assure you that if our users do guix pull and invisibly get a new
> R release,

They need to to ‘guix pull’ *and* ‘guix package -u’.

> their analyses will from time to time break. So we may want a simple
> way for them to back down to a previous release. So.. I am thinking it
> would make sense to keep previous versions of R in the recipe. What do
> others think?

I don’t use R myself; keeping a couple of R versions would be fine,
though R packages would always be built only against the latest R.

Ludo’.

Re: [PATCH] gnu: r: Update to 3.3.1.

[myglc2]

ludo@gnu.org (Ludovic Courtès) writes:

> myglc2 <myglc2@gmail.com> skribis:
>
>> I can assure you that if our users do guix pull and invisibly get a new
>> R release,
>
> They need to to ‘guix pull’ *and* ‘guix package -u’.
>
>> their analyses will from time to time break. So we may want a simple
>> way for them to back down to a previous release. So.. I am thinking it
>> would make sense to keep previous versions of R in the recipe. What do
>> others think?
>
> I don’t use R myself; keeping a couple of R versions would be fine,
> though R packages would always be built only against the latest R.

I had forgotten that you are also building many packages. Given that, an
update to the R guix recipe may break some packages. So all the packages
should probably be tested as part of an R recipe update. Do/did we do
that?

As regards keeping a couple of R versions, if we don't keep the
associated packages it will have limited usefulness, so I withdraw
the suggestion.

Re: [PATCH] gnu: r: Update to 3.3.1.

[Pjotr Prins]

On Sat, Jul 30, 2016 at 03:41:50PM -0400, myglc2 wrote:
> The workaround used by sysops where I work (hospital research lab) is to
> give notice of R upgrades and to make previous releases available for
> reference by ongoing projects. IMO, we should consider how the guix R
> recipe(s) might support a pattern of use like this.
> 
> I can assure you that if our users do guix pull and invisibly get a new
> R release, their analyses will from time to time break. So we may want a
> simple way for them to back down to a previous release. So.. I am
> thinking it would make sense to keep previous versions of R in the
> recipe. What do others think?

Hi George,

If bioconductor tests pass on a new R build they *should* work. Guix
is not the same mess that Debian and Fedora are.

Note, meanwhile, that a new R install does not remove the old packages
automatically. One way to work older versions is by using guix
profiles effectively. We introduced Unix modules with Guix, so a
module would point to a well tested and working profile. Just make
sure it does not get GC'd at some point.

Another way to work it is by using a checked out Guix source tree.
That is what we do for Genenetwork deployment. Guix 'pull' simply
represents a version of the source tree - with its combination of
packages. You can force a combination of package versions by keeping
track of your own tree. See 

  https://github.com/genenetwork/genenetwork2/blob/master/doc/README.org

In the near future I hope we get a version of guix pull which can
essentially achieve the same (i.e. a checked out version of the tree).

  guix pull HASH-tag

Pj.

Re: [PATCH] gnu: r: Update to 3.3.1.

[Roel Janssen]

> On Sat, Jul 30, 2016 at 03:41:50PM -0400, myglc2 wrote:
>> The workaround used by sysops where I work (hospital research lab) is to
>> give notice of R upgrades and to make previous releases available for
>> reference by ongoing projects. IMO, we should consider how the guix R
>> recipe(s) might support a pattern of use like this.
>> 
>> I can assure you that if our users do guix pull and invisibly get a new
>> R release, their analyses will from time to time break. So we may want a
>> simple way for them to back down to a previous release. So.. I am
>> thinking it would make sense to keep previous versions of R in the
>> recipe. What do others think?
>
> Hi George,
>
> If bioconductor tests pass on a new R build they *should* work. Guix
> is not the same mess that Debian and Fedora are.
>
> Note, meanwhile, that a new R install does not remove the old packages
> automatically. One way to work older versions is by using guix
> profiles effectively. We introduced Unix modules with Guix, so a
> module would point to a well tested and working profile. Just make
> sure it does not get GC'd at some point.
>
> Another way to work it is by using a checked out Guix source tree.
> That is what we do for Genenetwork deployment. Guix 'pull' simply
> represents a version of the source tree - with its combination of
> packages. You can force a combination of package versions by keeping
> track of your own tree. See 
>
>   https://github.com/genenetwork/genenetwork2/blob/master/doc/README.org
>
> In the near future I hope we get a version of guix pull which can
> essentially achieve the same (i.e. a checked out version of the tree).
>
>   guix pull HASH-tag

Hi George,

I think what you're looking for in your hospital research lab is what
Pjotr describes as a certain check-out of the Guix source tree.

From a scientific viewpoint you cannot say that the results of your data
analysis "will work with R version 3.3.0 or higher", but instead you can
only say "we achieved these results b using R version 3.3.0, with
extension X at version Y" (assuming these versions can be uniquely
identified to their source code).  This is what you can do with GNU
Guix.

You could explain how to reproduce your results by writing:  "When using
the recipes from GNU Guix at Git commit XXXX, we can construct a
software environment including R with version Y, with which we can
obtain the results we obtained."

So, using any newer Git commit ID is not guaranteed to work (but is
likely to work..).  The cool thing is, is that you can construct the
software environment from any particular time, as long as the source
tarballs are available.

In addition to the `per-user' profiles, you could use `per-pipeline' and
`per-group' profiles for users to "pin" a specific software environment
for doing the data analysis.  Users can then set the environment
variables in their shell to use that shared profile:
  export PATH=/path/to/profiles/per-pipeline/ngs/guix-profile/bin:$PATH
  export R_LIBS_SITE=/path/to/profiles/per-pipeline/ngs/guix-profile/site-library

Or by simply following the recommendations by GNU Guix:
  guix package --search-paths --profile=/path/to/profiles/per-profiles/ngs/guix-profile

I think upgrading is inevitable, so pinpointing to a specific set of
build recipes (tied to a commit ID) is a good way of maintaining a
stable software environment.

Do you think we can keep the latest versions in the upstream repository,
provided that you have a way of reverting or staying to the "old"
versions by either copying the 3.3.0 recipe to a local repository, or
pinpoint to an older Git commit?

Kind regards,
Roel Janssen

Re: [PATCH] gnu: r: Update to 3.3.1.

[myglc2]

Roel Janssen <roel@gnu.org> writes:

>> On Sat, Jul 30, 2016 at 03:41:50PM -0400, myglc2 wrote:
>>> I can assure you that if our users do guix pull and invisibly get a new
>>> R release, their analyses will from time to time break. So we may want a
>>> simple way for them to back down to a previous release. So.. I am
>>> thinking it would make sense to keep previous versions of R in the
>>> recipe. What do others think?

> I think what you're looking for in your hospital research lab is what
> Pjotr describes as a certain check-out of the Guix source tree.

But it is not simple to use. It is to technical an approach to appeal to
the medical researchers I have worked with.

> From a scientific viewpoint you cannot say that the results of your data
> analysis "will work with R version 3.3.0 or higher", but instead you can
> only say "we achieved these results b using R version 3.3.0, with
> extension X at version Y" (assuming these versions can be uniquely
> identified to their source code).

Actually R 'sessionInfo()' collects this at run time.

> The cool thing is, is that you can construct the software environment
> from any particular time, as long as the source tarballs are
> available.
>
> In addition to the `per-user' profiles, you could use `per-pipeline' and
> `per-group' profiles for users to "pin" a specific software environment
> for doing the data analysis.  Users can then set the environment
> variables in their shell to use that shared profile:
>   export PATH=/path/to/profiles/per-pipeline/ngs/guix-profile/bin:$PATH
>   export R_LIBS_SITE=/path/to/profiles/per-pipeline/ngs/guix-profile/site-library
>
> Or by simply following the recommendations by GNU Guix:
>   guix package --search-paths --profile=/path/to/profiles/per-profiles/ngs/guix-profile
>
> I think upgrading is inevitable, so pinpointing to a specific set of
> build recipes (tied to a commit ID) is a good way of maintaining a
> stable software environment.

Guix has marvelous raw tools to do anything. The question is how to make
it simple enough for someone that is basically an R user to take
advantage of them.  The challenge in guix R packaging is to consider R
patterns of use and determine how guix packate R to support them in a
way that is accessible to typical R users.

> Do you think we can keep the latest versions in the upstream repository,
> provided that you have a way of reverting or staying to the "old"
> versions by either copying the 3.3.0 recipe to a local repository, or
> pinpoint to an older Git commit?

Guix in general should have a scheme to decide which "golden oldies"
stay in the repository ;-)

In the meantime, after thinking about it some more I withdraw the
suggestion of multiple R version recipes (please see separate post).

But maybe you should test the existing guix R package recipes against
the new R version, if you have not already done so, before you update
the R recipe ;-)

Re: [PATCH] gnu: r: Update to 3.3.1.

[Pjotr Prins]

On Sun, Jul 31, 2016 at 01:49:00PM -0400, myglc2 wrote:
> > I think what you're looking for in your hospital research lab is what
> > Pjotr describes as a certain check-out of the Guix source tree.
> 
> But it is not simple to use. It is to technical an approach to appeal to
> the medical researchers I have worked with.

If it is that bad :) You have a choice of fixating the source (what I
do) or the binaries (see below).

> Guix has marvelous raw tools to do anything. The question is how to make
> it simple enough for someone that is basically an R user to take
> advantage of them.  The challenge in guix R packaging is to consider R
> patterns of use and determine how guix packate R to support them in a
> way that is accessible to typical R users.

What you need is a 'managed' environment for your users. My suggestion
is not to give guix daemon access to those users. Use Unix modules -
which I have packaged - to point them to a prepared profile. When they
want to use R, just make a profile. All modules do is set the PATHS,
as Roel described. Technology older than the Linux kernel :/

The 'manager' is the only one who will upgrade and test software to
run. That way you can do controlled upgrades. You can even have
multiple modules for different R's.

You are lucky you can run Guix daemon on your servers. Others build on
VMs and copy the files into shared storage. That also fixates the
binaries.

Pj.

Re: [PATCH] gnu: r: Update to 3.3.1.

[myglc2]

Pjotr Prins <pjotr.public12@thebird.nl> writes:

> On Sun, Jul 31, 2016 at 01:49:00PM -0400, myglc2 wrote:
>> > I think what you're looking for in your hospital research lab is what
>> > Pjotr describes as a certain check-out of the Guix source tree.
>> 
>> But it is not simple to use. It is to technical an approach to appeal to
>> the medical researchers I have worked with.
>
> If it is that bad :) You have a choice of fixating the source (what I
> do) or the binaries (see below).
>
>> Guix has marvelous raw tools to do anything. The question is how to make
>> it simple enough for someone that is basically an R user to take
>> advantage of them.  The challenge in guix R packaging is to consider R
>> patterns of use and determine how guix packate R to support them in a
>> way that is accessible to typical R users.
>
> What you need is a 'managed' environment for your users. My suggestion
> is not to give guix daemon access to those users. Use Unix modules -
> which I have packaged - to point them to a prepared profile. When they
> want to use R, just make a profile. All modules do is set the PATHS,
> as Roel described. Technology older than the Linux kernel :/
>
> The 'manager' is the only one who will upgrade and test software to
> run. That way you can do controlled upgrades. You can even have
> multiple modules for different R's.

I imagine that, in the spirit of guix, we also want a user to be able to
"help themself" instead of depending on a 'manager.' This would probably
require an additional R "package manager component" that is usable by a
manager or user. Such a thing would certainly showcase the unique
capabilities of guix.

> You are lucky you can run Guix daemon on your servers.

Actually this is only fantasy at the moment, but I am hopeful.

Re: [PATCH] gnu: r: Update to 3.3.1.

[Roel Janssen]

myglc2 writes:

> Pjotr Prins <pjotr.public12@thebird.nl> writes:
>
>> On Sun, Jul 31, 2016 at 01:49:00PM -0400, myglc2 wrote:
>>
>>> Guix has marvelous raw tools to do anything. The question is how to make
>>> it simple enough for someone that is basically an R user to take
>>> advantage of them.  The challenge in guix R packaging is to consider R
>>> patterns of use and determine how guix packate R to support them in a
>>> way that is accessible to typical R users.
>>
>> What you need is a 'managed' environment for your users. My suggestion
>> is not to give guix daemon access to those users. Use Unix modules -
>> which I have packaged - to point them to a prepared profile. When they
>> want to use R, just make a profile. All modules do is set the PATHS,
>> as Roel described. Technology older than the Linux kernel :/
>>
>> The 'manager' is the only one who will upgrade and test software to
>> run. That way you can do controlled upgrades. You can even have
>> multiple modules for different R's.
>
> I imagine that, in the spirit of guix, we also want a user to be able to
> "help themself" instead of depending on a 'manager.' This would probably
> require an additional R "package manager component" that is usable by a
> manager or user. Such a thing would certainly showcase the unique
> capabilities of guix.

And users can!  The "software manager" can provide "supported" profiles,
and the users can still create their own software environment.  Then
when things break, users are one command away from switching to a
working environment (provided by the manager).  This safety net provides
a confidence to play around even more..

The software manager can install packages from his own custom recipes,
separate from whatever `guix pull' provides by setting the
GUIX_PACKAGE_PATH variable.  Users do not need to know about it, if they
don't want to.

Kind regards,
Roel Janssen

Re: [PATCH] gnu: r: Update to 3.3.1.

[myglc2]

Roel Janssen <roel@gnu.org> writes:

> myglc2 writes:
>
>> Pjotr Prins <pjotr.public12@thebird.nl> writes:
>>
>>> On Sun, Jul 31, 2016 at 01:49:00PM -0400, myglc2 wrote:
>>>
>>>> Guix has marvelous raw tools to do anything. The question is how to make
>>>> it simple enough for someone that is basically an R user to take
>>>> advantage of them.  The challenge in guix R packaging is to consider R
>>>> patterns of use and determine how guix packate R to support them in a
>>>> way that is accessible to typical R users.
>>>
>>> What you need is a 'managed' environment for your users. My suggestion
>>> is not to give guix daemon access to those users. Use Unix modules -
>>> which I have packaged - to point them to a prepared profile. When they
>>> want to use R, just make a profile. All modules do is set the PATHS,
>>> as Roel described. Technology older than the Linux kernel :/
>>>
>>> The 'manager' is the only one who will upgrade and test software to
>>> run. That way you can do controlled upgrades. You can even have
>>> multiple modules for different R's.
>>
>> I imagine that, in the spirit of guix, we also want a user to be able to
>> "help themself" instead of depending on a 'manager.' This would probably
>> require an additional R "package manager component" that is usable by a
>> manager or user. Such a thing would certainly showcase the unique
>> capabilities of guix.
>
> And users can!  The "software manager" can provide "supported" profiles,
> and the users can still create their own software environment.  Then
> when things break, users are one command away from switching to a
> working environment (provided by the manager).  This safety net provides
> a confidence to play around even more..
>
> The software manager can install packages from his own custom recipes,
> separate from whatever `guix pull' provides by setting the
> GUIX_PACKAGE_PATH variable.  Users do not need to know about it, if they
> don't want to.

Yes I get that it is possible now... but I feel strongly that the
typical R user will be overwhealmed by the guix "wheels and levers".

So I resort to fantasizing a mythical additional guix-specific
component, an 'R "package manager component"'. The idea is to exploit
guix capabilities to make this much more tractable for an average R user
and much more convenient for the sysadmin in charge of providing support
to R users.

Such an 'R "package manager component"' would make the argument for guix
isntallation very compelling for medical research labs.

Re: [PATCH] gnu: r: Update to 3.3.1.

[Roel Janssen]

myglc2 writes:

> Roel Janssen <roel@gnu.org> writes:
>
>> myglc2 writes:
>>
>>> Pjotr Prins <pjotr.public12@thebird.nl> writes:
>>>
>>>> On Sun, Jul 31, 2016 at 01:49:00PM -0400, myglc2 wrote:
>>>>
>>>>> Guix has marvelous raw tools to do anything. The question is how to make
>>>>> it simple enough for someone that is basically an R user to take
>>>>> advantage of them.  The challenge in guix R packaging is to consider R
>>>>> patterns of use and determine how guix packate R to support them in a
>>>>> way that is accessible to typical R users.
>>>>
>>>> What you need is a 'managed' environment for your users. My suggestion
>>>> is not to give guix daemon access to those users. Use Unix modules -
>>>> which I have packaged - to point them to a prepared profile. When they
>>>> want to use R, just make a profile. All modules do is set the PATHS,
>>>> as Roel described. Technology older than the Linux kernel :/
>>>>
>>>> The 'manager' is the only one who will upgrade and test software to
>>>> run. That way you can do controlled upgrades. You can even have
>>>> multiple modules for different R's.
>>>
>>> I imagine that, in the spirit of guix, we also want a user to be able to
>>> "help themself" instead of depending on a 'manager.' This would probably
>>> require an additional R "package manager component" that is usable by a
>>> manager or user. Such a thing would certainly showcase the unique
>>> capabilities of guix.
>>
>> And users can!  The "software manager" can provide "supported" profiles,
>> and the users can still create their own software environment.  Then
>> when things break, users are one command away from switching to a
>> working environment (provided by the manager).  This safety net provides
>> a confidence to play around even more..
>>
>> The software manager can install packages from his own custom recipes,
>> separate from whatever `guix pull' provides by setting the
>> GUIX_PACKAGE_PATH variable.  Users do not need to know about it, if they
>> don't want to.
>
> Yes I get that it is possible now... but I feel strongly that the
> typical R user will be overwhealmed by the guix "wheels and levers".
>
> So I resort to fantasizing a mythical additional guix-specific
> component, an 'R "package manager component"'. The idea is to exploit
> guix capabilities to make this much more tractable for an average R user
> and much more convenient for the sysadmin in charge of providing support
> to R users.
>
> Such an 'R "package manager component"' would make the argument for guix
> isntallation very compelling for medical research labs.

Right.  I see we can do better indeed.

Kind regards,
Roel Janssen

Re: [PATCH] gnu: r: Update to 3.3.1.

[Ricardo Wurmus]

myglc2 <myglc2@gmail.com> writes:

> I imagine that, in the spirit of guix, we also want a user to be able to
> "help themself" instead of depending on a 'manager.' This would probably
> require an additional R "package manager component" that is usable by a
> manager or user. Such a thing would certainly showcase the unique
> capabilities of guix.

We have importers that generate Guix package expressions from CRAN or
Bioconductor.  Taken one step further we might have a transparent
wrapper to quickly install any version of a package even if it is not
yet part of Guix.

This might work like this:

* user issues “guix package --install --via cran gdtools@0.0.3”

* Guix runs the CRAN importe to recursively generate package expressions
  for “gdtools” at the given version.  The package does not have to
  exist in Guix yet.  The generated package expressions are cached
  somewhere (e.g. ~/.cache/guix/…genomation…).

* Guix loads the newly imported package expressions from
  ~/.cache/guix/…genomation… and tries to build them.

* Optionally, users can take the imported expressions, clean them up and
  send them as patches to us :)

This obviously depends on the quality of the importers, but I think it
wouldn’t take much effort to make this a reality.  The main problem with
offering magic like this is that it must be reliable or else it would
just be frustrating users.

At least for CRAN I think this is feasible.  (Bioconductor is a
different beast as versions would need to be given as Bioconductor SVN
revisions, which breaks the package boundaries.)

~~ Ricardo

Re: [PATCH] gnu: r: Update to 3.3.1.

[myglc2]

Ricardo Wurmus <rekado@elephly.net> writes:

> myglc2 <myglc2@gmail.com> writes:
>
>> I imagine that, in the spirit of guix, we also want a user to be able to
>> "help themself" instead of depending on a 'manager.' This would probably
>> require an additional R "package manager component" that is usable by a
>> manager or user. Such a thing would certainly showcase the unique
>> capabilities of guix.
>
> We have importers that generate Guix package expressions from CRAN or
> Bioconductor.  Taken one step further we might have a transparent
> wrapper to quickly install any version of a package even if it is not
> yet part of Guix.
>
> This might work like this:
>
> * user issues “guix package --install --via cran gdtools@0.0.3”
>
> * Guix runs the CRAN importe to recursively generate package expressions
>   for “gdtools” at the given version.  The package does not have to
>   exist in Guix yet.  The generated package expressions are cached
>   somewhere (e.g. ~/.cache/guix/…genomation…).
>
> * Guix loads the newly imported package expressions from
>   ~/.cache/guix/…genomation… and tries to build them.
>
> * Optionally, users can take the imported expressions, clean them up and
>   send them as patches to us :)
>
> This obviously depends on the quality of the importers, but I think it
> wouldn’t take much effort to make this a reality.

Could this approach also support earlier R versions? Reasons one would
want this:

- one comes back to a project after sebatical. Because new R versions
  have been installed the analysis gives different results and/or is
  broken.
  
- one needs to re-run an analysis to reproduce a result or to do
  sensitivity studies.

In such cases it is far preferable, but usually problematic, to
reconstruct the environment in which the anaysis was run. If guix could
be used to do this, it would be a compelling argument for adoption in a
research lab situation.

> The main problem with offering magic like this is that it must be
> reliable or else it would just be frustrating users.

Agreed. But any experienced R user knows that package installation can
be a drama. So it does not have to be perfect to match or exceed user
expectaions. If the user's alternatives are ...

1) wait for sysadmin to do it

2) give this a shot

... most users will happily try pushing this button ;-)

- George

Re: [PATCH] gnu: r: Update to 3.3.1.

[Ricardo Wurmus]

myglc2 <myglc2@gmail.com> writes:

> Ricardo Wurmus <rekado@elephly.net> writes:
>
>> myglc2 <myglc2@gmail.com> writes:
>>
>>> I imagine that, in the spirit of guix, we also want a user to be able to
>>> "help themself" instead of depending on a 'manager.' This would probably
>>> require an additional R "package manager component" that is usable by a
>>> manager or user. Such a thing would certainly showcase the unique
>>> capabilities of guix.
>>
>> We have importers that generate Guix package expressions from CRAN or
>> Bioconductor.  Taken one step further we might have a transparent
>> wrapper to quickly install any version of a package even if it is not
>> yet part of Guix.
>>
>> This might work like this:
>>
>> * user issues “guix package --install --via cran gdtools@0.0.3”
>>
>> * Guix runs the CRAN importe to recursively generate package expressions
>>   for “gdtools” at the given version.  The package does not have to
>>   exist in Guix yet.  The generated package expressions are cached
>>   somewhere (e.g. ~/.cache/guix/…genomation…).
>>
>> * Guix loads the newly imported package expressions from
>>   ~/.cache/guix/…genomation… and tries to build them.
>>
>> * Optionally, users can take the imported expressions, clean them up and
>>   send them as patches to us :)
>>
>> This obviously depends on the quality of the importers, but I think it
>> wouldn’t take much effort to make this a reality.
>
> Could this approach also support earlier R versions? Reasons one would
> want this:
>
> - one comes back to a project after sebatical. Because new R versions
>   have been installed the analysis gives different results and/or is
>   broken.
>   
> - one needs to re-run an analysis to reproduce a result or to do
>   sensitivity studies.
>
> In such cases it is far preferable, but usually problematic, to
> reconstruct the environment in which the anaysis was run. If guix could
> be used to do this, it would be a compelling argument for adoption in a
> research lab situation.

At the institute where I work and where Guix is deployed we never
upgrade users packages centrally.  Scientists use per-project manifests
to keep a list of packages they want to have installed for their current
work.

After returning from vacation their software is still the same as only
they upgrade things.

We also encourage them to record the git hash of both package
repositories we use (one for Guix upstream and another for our custom
Guix BIMSB repo) at the time they instantiate a manifest.  We provide a
wrapper that essentially just runs “git describe” on the two repos.

Given the manifest and the two git hashes a user can reproduce the exact
same binary state of a profile.  This is more granular than thinking in
terms of version numbers as this captures the complete graph of
dependencies.

>> The main problem with offering magic like this is that it must be
>> reliable or else it would just be frustrating users.
>
> Agreed. But any experienced R user knows that package installation can
> be a drama. So it does not have to be perfect to match or exceed user
> expectaions. If the user's alternatives are ...
>
> 1) wait for sysadmin to do it
>
> 2) give this a shot
>
> ... most users will happily try pushing this button ;-)

This has also been my experience here at the MDC.  Some are already
using the CRAN importer themselves to install packages via Guix that I
haven’t packaged for them yet.  (Admittedly, as bioinformaticians they
are likely more comfortable with the command line than scientists in
other fields.)

~~ Ricardo

Re: [PATCH] gnu: r: Update to 3.3.1.

[Ludovic Courtès]

Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

> This obviously depends on the quality of the importers, but I think it
> wouldn’t take much effort to make this a reality.  The main problem with
> offering magic like this is that it must be reliable or else it would
> just be frustrating users.

It’s indeed doable (we just need to generate a <package> on the fly,
which is essentially what importers do), and was part of the plan.

But as you note, the likely problem is that some packages won’t build,
for instance because we have incomplete or inaccurate dependency
information, and may be poorly integrated.

Nevertheless it’s worth trying!

Ludo’.

Re: [PATCH] gnu: r: Update to 3.3.1.

[Pjotr Prins]

On Mon, Aug 01, 2016 at 10:02:05PM +0200, Ricardo Wurmus wrote:
> We have importers that generate Guix package expressions from CRAN or
> Bioconductor.  Taken one step further we might have a transparent
> wrapper to quickly install any version of a package even if it is not
> yet part of Guix.
> 
> This might work like this:
> 
> * user issues “guix package --install --via cran gdtools@0.0.3”
> 
> * Guix runs the CRAN importe to recursively generate package expressions
>   for “gdtools” at the given version.  The package does not have to
>   exist in Guix yet.  The generated package expressions are cached
>   somewhere (e.g. ~/.cache/guix/…genomation…).
> 
> * Guix loads the newly imported package expressions from
>   ~/.cache/guix/…genomation… and tries to build them.
> 
> * Optionally, users can take the imported expressions, clean them up and
>   send them as patches to us :)
> 
> This obviously depends on the quality of the importers, but I think it
> wouldn’t take much effort to make this a reality.  The main problem with
> offering magic like this is that it must be reliable or else it would
> just be frustrating users.
> 
> At least for CRAN I think this is feasible.  (Bioconductor is a
> different beast as versions would need to be given as Bioconductor SVN
> revisions, which breaks the package boundaries.)

Another thing to keep in mind is that CRAN has the nasty habit of
moving older source packages to archive directories. Some packages are
removed altogether. We can/should make use of Guix' own source cache.

Pj.

Re: [PATCH] gnu: r: Update to 3.3.1.

[Ricardo Wurmus]

Pjotr Prins <pjotr.public12@thebird.nl> writes:

> On Mon, Aug 01, 2016 at 10:02:05PM +0200, Ricardo Wurmus wrote:
>> We have importers that generate Guix package expressions from CRAN or
>> Bioconductor.  Taken one step further we might have a transparent
>> wrapper to quickly install any version of a package even if it is not
>> yet part of Guix.
>> 
>> This might work like this:
>> 
>> * user issues “guix package --install --via cran gdtools@0.0.3”
>> 
>> * Guix runs the CRAN importe to recursively generate package expressions
>>   for “gdtools” at the given version.  The package does not have to
>>   exist in Guix yet.  The generated package expressions are cached
>>   somewhere (e.g. ~/.cache/guix/…genomation…).
>> 
>> * Guix loads the newly imported package expressions from
>>   ~/.cache/guix/…genomation… and tries to build them.
>> 
>> * Optionally, users can take the imported expressions, clean them up and
>>   send them as patches to us :)
>> 
>> This obviously depends on the quality of the importers, but I think it
>> wouldn’t take much effort to make this a reality.  The main problem with
>> offering magic like this is that it must be reliable or else it would
>> just be frustrating users.
>> 
>> At least for CRAN I think this is feasible.  (Bioconductor is a
>> different beast as versions would need to be given as Bioconductor SVN
>> revisions, which breaks the package boundaries.)
>
> Another thing to keep in mind is that CRAN has the nasty habit of
> moving older source packages to archive directories. Some packages are
> removed altogether. We can/should make use of Guix' own source cache.

We already get around the first of these two problems by specifying a
list of source URLs via “cran-uri”: the first is the plain URL of the
latest tarball, the second is a URL from the archive.  This allows us to
continue to build CRAN packages even when their source tarballs are
moved to the archive.

For Bioconductor this doesn’t work (as they have no archives at all) and
we’ll have to switch to building from SVN eventually.

~~ Ricardo

Re: [PATCH] gnu: r: Update to 3.3.1.

[myglc2]

Pjotr Prins <pjotr.public12@thebird.nl> writes:

> On Sat, Jul 30, 2016 at 03:41:50PM -0400, myglc2 wrote:
>> The workaround used by sysops where I work (hospital research lab) is to
>> give notice of R upgrades and to make previous releases available for
>> reference by ongoing projects. IMO, we should consider how the guix R
>> recipe(s) might support a pattern of use like this.
>> 
>> I can assure you that if our users do guix pull and invisibly get a new
>> R release, their analyses will from time to time break. So we may want a
>> simple way for them to back down to a previous release. So.. I am
>> thinking it would make sense to keep previous versions of R in the
>> recipe. What do others think?

> Note, meanwhile, that a new R install does not remove the old packages
> automatically. One way to work older versions is by using guix
> profiles effectively. We introduced Unix modules with Guix, so a
> module would point to a well tested and working profile. Just make
> sure it does not get GC'd at some point.

This functionality could be adequate in some situations if it can be
made simple to use. Some questions to answer...

What do you mean by "Unix modules"?

How does one "make sure it does not get GC'd"?

What happens when a user wants something else (e.g., not R) updated?

> Another way to work it is by using a checked out Guix source tree.

This is not simple and is beyond the capability of the medical
researchers I have met.

> In the near future I hope we get a version of guix pull which can
> essentially achieve the same (i.e. a checked out version of the tree).
>
>   guix pull HASH-tag

This comes the closet to being a "simple way for them to back down to a
previous release". It has merit independent of R.

Re: [PATCH] gnu: r: Update to 3.3.1.

[Roel Janssen]

myglc2 writes:

> Pjotr Prins <pjotr.public12@thebird.nl> writes:
>
>> On Sat, Jul 30, 2016 at 03:41:50PM -0400, myglc2 wrote:
>>> The workaround used by sysops where I work (hospital research lab) is to
>>> give notice of R upgrades and to make previous releases available for
>>> reference by ongoing projects. IMO, we should consider how the guix R
>>> recipe(s) might support a pattern of use like this.
>>> 
>>> I can assure you that if our users do guix pull and invisibly get a new
>>> R release, their analyses will from time to time break. So we may want a
>>> simple way for them to back down to a previous release. So.. I am
>>> thinking it would make sense to keep previous versions of R in the
>>> recipe. What do others think?
>
>> Note, meanwhile, that a new R install does not remove the old packages
>> automatically. One way to work older versions is by using guix
>> profiles effectively. We introduced Unix modules with Guix, so a
>> module would point to a well tested and working profile. Just make
>> sure it does not get GC'd at some point.
>
> This functionality could be adequate in some situations if it can be
> made simple to use. Some questions to answer...
>
> What do you mean by "Unix modules"?

I think Pjotr means environment variables.

> How does one "make sure it does not get GC'd"?

Keep it in a profile linked to the store.  (So once you've installed
a version in a profile, Guix will not GC it as long as you haven't
removed the profile, or the programs in the profiles and its
accompanying generations).

> What happens when a user wants something else (e.g., not R) updated?

He can just to `guix package --upgrade=<specific-package>'.

>> Another way to work it is by using a checked out Guix source tree.
>
> This is not simple and is beyond the capability of the medical
> researchers I have met.

I agree.  However, when upgrading packages, they can be careful not to
upgrade a package.  You can do that by `guix package --upgrade
--do-not-upgrade=r'.

This may seem too difficult as well, but let's consider the alternative:
Not upgrading packages upstream.  That's what we can already do
downstream (simply don't run `guix pull').  I like having the latest
versions available in GNU Guix, and when I need an older version, I can
find it in the version control system.

Kind regards,
Roel Janssen

Re: [PATCH] gnu: r: Update to 3.3.1.

[Andreas Enge]

Hello,

On Sat, Jul 30, 2016 at 04:36:22PM +0200, Roel Janssen wrote:
> Is it OK to upgrade?

maybe wait until after the core-updates merge?

Andreas

Re: [PATCH] gnu: r: Update to 3.3.1.

[Ricardo Wurmus]

Roel Janssen <roel@gnu.org> writes:

> Dear Guix,
>
> With the following patch I would like to update R to 3.3.1.  I tested
> with R packages I regularly use and everything seems to work just fine.
>
> Is it OK to upgrade?

In principle: yes.

I’ll see if I can make updates to all R packages later, once your
R upgrade is in master.

~~ Ricardo