Making the case for GNU Guix ... advice sought

Making the case for GNU Guix ... advice sought

[Cook, Malcolm]

Hi,

I have been asked to write up an argument for the advantages that GNU Guix confers for deploying linux software, especially scientific computing and including bioinformatics software.
	
To that end I have written up this document 

	https://github.com/malcook/sce/blob/master/MakingTheCase.org

and I welcome all criticism, suggestions, observations, omissions, flames, what-have-you.    Please feel free to discuss in this thread, via private email, pull requests, as you see fit.   

I hope to incorporate the "good parts" of this discussion back into the document and welcome anyone to use as they see fit (that would be a nice outcome!).

I have some notes contrasting GNU Guix with other similar/related tool-sets (modules, lmod, homebrew, spack, easyBuild, rolling rpms), as well as our current practices (sort of hybrid of rpms, bastard son of homebrew, and "just wedge it in there").  However I am not now intent upon setting up such a contrast, rather, I hope to focus on the advantages of GNU Guix in general.

Some of my comments may be naïve/mis-informed as my experience with guix so far has been small personal testing deployments, and I have not worked out all the details of deploying " Using a shared Guix store" as discussed here: http://guix-devel.gnu.narkive.com/tk2nRunF/using-a-shared-guix-store-was-re-bio-packaging-testing-out-guix - about which I also welcome further comments, suggestions, and/or warnings.

Thanks,

Malcolm Cook
 

Re: Making the case for GNU Guix ... advice sought

[Ricardo Wurmus]

Cook, Malcolm <MEC@stowers.org> writes:

> I have been asked to write up an argument for the advantages that GNU
> Guix confers for deploying linux software, especially scientific
> computing and including bioinformatics software.
> 	
> To that end I have written up this document 
>
> 	https://github.com/malcook/sce/blob/master/MakingTheCase.org

Nice!

Here’s a first correction for a mistake that I’m responsible for: the
number of bioinfo packages is actually closer to 114 (not 54).  Guix web
was misconfigured on that host and it would show the status of a very,
very outdated version of Guix rather than the latest and greatest.

(Note that a very small number of the packages listed there are released
under restrictive “academic only” licenses or have undeclared licenses;
the license field says “non-free” or “undeclared”.  Having them on this
page does not mean I endorse the use of these tools and users should be
careful to check the license.)

> I have some notes contrasting GNU Guix with other similar/related
> tool-sets (modules, lmod, homebrew, spack, easyBuild, rolling rpms),
> as well as our current practices (sort of hybrid of rpms, bastard son
> of homebrew, and "just wedge it in there").  However I am not now
> intent upon setting up such a contrast, rather, I hope to focus on the
> advantages of GNU Guix in general.

I’d also like to point you at https://hal.inria.fr/hal-01161771/en, in
case you haven’t read it yet.  It includes a very general comparison
with tools like environment modules, spack, and easyBuild with a focus
on reproducibility.

> + Guix detects and prohibits program name collisions; loading
>   conflicting packages into a users environment is _impossible_.  For
>   example, this prevents the ambiguity and associated error that can
>   arise when two packages define different programs by the same name
>   and both are placed in the user's execution PATH.

Well... I don’t know if this is worth pointing out.  Guix does detect
collisions but it doesn’t do anything intelligent here, maybe because
there isn’t much that can be done when a collision happens.  You mention
support for multiple profiles later, and I think that maybe this could
be merged.  I found collision detection to be not so useful because it
happens *during* profile generation, not *before* I commit to installing
a package.

> + Guix packages naturally extend to include all package resources,
>   including man pages, libraries, as well as binaries.  This is a
>   result of their using the underlying build engine's (i.e. gnu make)
>   installation targets.

Is there any package manager that does not handle this?

> + Guix packages, being independent of host on which they are built,
>   can be downloaded already built by upstream servers known as
>   [[https://www.gnu.org/software/guix/manual/html_node/Substitutes.html#Substitutes][substitues]], with the assurance of their being bit-level identical
>   to the results of the generally longer process of configuration and
>   compilation on local servers.

Building packages in isolated environments is a necessary requirement
for bit-reproducibility, but it is not sufficient in itself.  To speak
of “assurance” is maybe a bit strong.

> + Guix packages, though dependent upon the machine architecture, are
>   independent on the linux distribution, or its version.  Thus, for
>   example, packages built under CentOS 6.5 on x86_64 will run under
>   operating system on x86_64, for example, CentOS 7.x and or Ubuntu
>   15.10 or 14.04.

This is true and it’s a great feature.  We’re using the very same Guix
store on various subversions of CentOS 6.x, a wide range of Ubuntus, and
Fedora.  “Linux distribution” is confusing, though.  I’d write
“GNU/Linux distribution”.  There *are* some kernel requirements, so the
version of Linux itself (i.e. the kernel version) does matter up to a
point for some of the advanced features of Guix (such as container
support).

> + Guix allows for unprivileged package management; users do not need
>   special elevated privileges (root or sudo) to create custom
>   environment profiles.  Especially noteworthy is that this allows
>   for rationally sharing package management as a distributed
>   responsibility.  This includes installing new site-available
>   applications (for those available in the guix repositories).  Thus,
>   if one user observes that a new release of a site-installed
>   application has become available, that user can safely install the
>   upgrade centrally and immediately start using it, without effecting
            ^—— what does this mean?  At the centre of ... what?

>   any other user's environment; without any further coordination,
>   when another user _is_ ready to adopt the upgrade, they will find
>   that installation is unnecessary, as it has already occurred.

I would clearly separate building and installing.  Installation is the
act of creating a new profile generation that contains a link to a
previously built (or substituted) item in the store.  Building (or
substituting) is what happens only once because of the purely functional
properties of the Guix package “functions”.

> + Guix provides a set of [[https://www.gnu.org/software/guix/manual/guix.html#Build-Systems][build systems]] providing support for language
>   specific package management systems, including R, perl, python,
>   ruby, haskell, and emacs.  This should allow a single approach for
>   managing computing environments for each of these languages/tools,
>   as opposed to needing to master the ideosyncracies of multiple
>   approaches, i.e. perlbrew for perl, pyenv or pythonbrew for python,
>   rbenv for ruby, etc.

Build systems are unrelated to “guix environment” or “rbenv”,
“virtualenv” and the like.  A build system is just a generalisation of
the steps that need to performed to build something.  For GNU-style
packages that’s

    ./configure --prefix=/something && make && make install

for Python it’s something like

    python setup.py install

etc.  The build systems make *packaging* easier as we no longer need to
express the build procedure for R packages in terms of the GNU build
system (which would be inappropriate).  This gives us the power of
abstraction, something that package managers like “conda” do not aim to
provide — there you need to ship a script along with the metadata file,
which is then run to build the package.  The script may need to do
little more than

    ./configure && make && make install

but it is a much less principled approach as bash has hardly the means
to provide for clear and easy abstraction.  (And to have to run
arbitrary shell scripts without any sort of isolation is not very
encouraging.)

The fact that build processes run as unprivileged, dedicated build users
is another feature: you don’t need to be afraid that the build script
breaks your precious home directory.  (See
https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/commit/a047be85247755cdbe0acce6f1dafc8beb84f2ac
for a shell script bug that isn’t all that rare and could really spoil
your day if you’re running a script like that without isolation.)

> + Guix control of build processes down to level of
>   binary-compatibility, along with the management tools for
>   environment profiles can provide the basis for improvements in
>   reproducible pipelines, such as are begining to appear in
>   computational centers of excellence.

This sounds a bit confused.  When applying functional package management
ideas to a package and *all* of its dependencies recursively, you end up
with a directed acyclic graph (loops are “unrolled” by bootstrapping
with existing packages where possible) that is rooted in a very small
number of essential bootstrap binaries.  Together with build-time
isolation we get very close to a reproducible software stack.  The key
idea here, though, is functional package management because without it
build-time isolation wouldn’t be of much use.

> + Guix provides commands to
>   [[https://www.gnu.org/software/guix/manual/guix.html#Emacs-Commands][display
>   package information]] which can be used to automate the production
>   and publishing of software package catalogs.  Such catalogs may then
>   be shared, printed, emailed, or embedded into community-visible web
>   pages as various means of advertising package availability to the
>   research community.

Guix is a library, which allows for the Emacs interface and the Guix web
interface (which the MDC uses to display what software we have available
in Guix).

> + Guix package specifications, being written in [[https://www.gnu.org/software/guix/manual/guix.html#Defining-Packages][Guile/scheme]], do not
>   depend upon the users SHELL (i.e. guix works equally well with zsh,
>   bash, tcsh, etc) (TODO: confirm).

The user’s shell does not matter, but this is nothing to do with being
written in Scheme.  The Guix daemon spawns a shell on its own that does
not depend on the user’s environment.

> + Guix has a future - it is the package manager for the new
>   GNU-backed linux distribution, [[https://www.gnu.org/software/guix/][GuixSD]].

“linux distribution” ...  GuixSD provides a variant of the GNU system
(which by default happens to use Linux as its kernel).

> + Guix is [[https://www.gnu.org/software/guix/manual/][well documented]].

Note that the manual on the website is somewhat outdated.  It’s the
manual for the latest 0.9.0 release (which is already quite some time
ago).  The latest version is only available in the git repository.
Maybe we should change this.

 > + Guix community has already prepared [[https://www.gnu.org/software/guix/packages/][many recipes]], of which
 >   currently [[http://guix.mdc-berlin.de/packages?/?search=bioinfo][54 are bioinformatics]] packages.

As I wrote above: it’s 114 (thanks for prompting me to fix the
configuration error that made it show 54 packages only).

> + Guix packaging is relatively easy to learn. It is reasonably
>   documented and there are `lint` style tools that check recipes for
>   being well-structured; they identify common errors in package
>   specification.

We also have importers (some with great, others with good results) so
that often there isn’t much work to be done at all.  We have importers
for CRAN and bioconductor, which worked pretty well for me; also the
hackage importer is great and it saved me a lot of time when I packaged
all missing dependencies for pandoc in about an afternoon.

The importers are really very useful and I’d definitely mention them.

> + Guix development is open source.  It is open to input from all
>   community members.

It’s free software ;)

Maybe it’s also good to mention that you do not need to rely on Guix
upstream to add packages.  It is trivial to use custom packages with
GUIX_PACKAGE_PATH.

> + Guix exposes the actual system calls to the package developer.

What does this mean?

~~ Ricardo

Re: Making the case for GNU Guix ... advice sought

[Leo Famulari]

On Tue, Feb 09, 2016 at 04:03:13PM +0100, Ricardo Wurmus wrote:
> 
> Cook, Malcolm <MEC@stowers.org> writes:
> > + Guix development is open source.  It is open to input from all
> >   community members.
> 
> It’s free software ;)
> 
> Maybe it’s also good to mention that you do not need to rely on Guix
> upstream to add packages.  It is trivial to use custom packages with
> GUIX_PACKAGE_PATH.

It can't be overstated *how* trivial it is. I've found that users of
mutable-state package managers just don't believe me when I tell them.
It's even easy to have out-of-tree packages, pulled in with
GUIX_PACKAGE_PATH, that depend on in-tree packages or package updates
that you haven't upstreamed yet.

Re: Making the case for GNU Guix ... advice sought

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

> On Tue, Feb 09, 2016 at 04:03:13PM +0100, Ricardo Wurmus wrote:
>> 
>> Cook, Malcolm <MEC@stowers.org> writes:
>> > + Guix development is open source.  It is open to input from all
>> >   community members.
>> 
>> It’s free software ;)
>> 
>> Maybe it’s also good to mention that you do not need to rely on Guix
>> upstream to add packages.  It is trivial to use custom packages with
>> GUIX_PACKAGE_PATH.
>
> It can't be overstated *how* trivial it is. I've found that users of
> mutable-state package managers just don't believe me when I tell them.
> It's even easy to have out-of-tree packages, pulled in with
> GUIX_PACKAGE_PATH, that depend on in-tree packages or package updates
> that you haven't upstreamed yet.

I’ve been thinking that we should add a “Defining Package Variants”
section beneath or after the “Defining Packages” node in the manual,
showing how to ‘inherit’ from an existing package, etc.

Any takers?  :-)

Ludo’.

Re: Making the case for GNU Guix ... advice sought

[Ludovic Courtès]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> I’d also like to point you at https://hal.inria.fr/hal-01161771/en, in
> case you haven’t read it yet.  It includes a very general comparison
> with tools like environment modules, spack, and easyBuild with a focus
> on reproducibility.

One of the things that has changed since this paper is the new --input
flag which allows you, from the command-line, to rewrite the dependency
tree of a package by replacing one package with another one (something
that’s relatively easy to do programmatically, but Spack showed that
it’s occasionally convenient to have it at the command line.)

Ludo’.

Re: Making the case for GNU Guix ... advice sought

[Ludovic Courtès]

"Cook, Malcolm" <MEC@stowers.org> skribis:

> I have been asked to write up an argument for the advantages that GNU Guix confers for deploying linux software, especially scientific computing and including bioinformatics software.
> 	
> To that end I have written up this document 
>
> 	https://github.com/malcook/sce/blob/master/MakingTheCase.org

Good idea, and nice work!

Perhaps it would be worth mentioning ‘guix environment’ and
‘--search-paths’.

But really, I don’t have much to add to Ricardo’s detailed review.  :-)

Thank you,
Ludo’.

RE: Making the case for GNU Guix ... advice sought

[Cook, Malcolm]

Hi,

Thanks everyone for the many valuable suggestions and education.

Next week I intend to incorporate the suggested improvements as best I can and follow up with an edit that I will use in-house, and share back out via git-hub.

Though the document is aimed at my particular internal "project justification", I am trying to keep it generic, and will welcome suggestions for further reshaping it to be of more general value should there be such interest from y'all.

GoGuix,

Malcolm

RE: Making the case for GNU Guix ... advice sought

[Cook, Malcolm]

Hi,

I've tried to incorporate your collective feedback to https://github.com/malcook/sce/blob/master/MakingTheCase.org which this commit

	https://github.com/malcook/sce/commit/917b55e1a0a9d8a5b79aa43b2a751db603cca34d?diff=split

You can expect another round of edits once I get clear about some of Ricardo's thoughtful & detailed comments (to which I have just now replied with some add-on questions).

Thanks everyone, this is very helpful to me, 

~Malcolm

 > -----Original Message-----
 > From: guix-devel-bounces+mec=stowers.org@gnu.org [mailto:guix-devel-
 > bounces+mec=stowers.org@gnu.org] On Behalf Of Cook, Malcolm
 > Sent: Wednesday, February 10, 2016 10:02 AM
 > To: 'Ludovic Courtès' <ludo@gnu.org>; 'Ricardo Wurmus'
 > <ricardo.wurmus@mdc-berlin.de>; 'Pjotr Prins' <pjotr2016@thebird.nl>; 'Leo
 > Famulari' <leo@famulari.name>
 > Cc: 'Guix-devel' <guix-devel@gnu.org>; 'bio-packaging@mailman.open-
 > bio.org' <bio-packaging@mailman.open-bio.org>
 > Subject: RE: Making the case for GNU Guix ... advice sought
 > 
 > Hi,
 > 
 > Thanks everyone for the many valuable suggestions and education.
 > 
 > Next week I intend to incorporate the suggested improvements as best I can
 > and follow up with an edit that I will use in-house, and share back out via git-
 > hub.
 > 
 > Though the document is aimed at my particular internal "project justification",
 > I am trying to keep it generic, and will welcome suggestions for further
 > reshaping it to be of more general value should there be such interest from
 > y'all.
 > 
 > GoGuix,
 > 
 > Malcolm

RE: Making the case for GNU Guix ... advice sought

[Cook, Malcolm]

Hi Ricardo,

Great specifics.  If you can help me solve a few of my remaining questions, in-line below, I'd be much obliged....

> Cook, Malcolm <MEC@stowers.org> writes:
 > 
 > > I have been asked to write up an argument for the advantages that GNU
 > > Guix confers for deploying linux software, especially scientific
 > > computing and including bioinformatics software.
 > >
 > > To that end I have written up this document
 > >
 > > 	https://github.com/malcook/sce/blob/master/MakingTheCase.org
 > 
 > Nice!

Thanks.  I hope it helps fellow-travelers later...

 > Here’s a first correction for a mistake that I’m responsible for: the
 > number of bioinfo packages is actually closer to 114 (not 54).  Guix web
 > was misconfigured on that host and it would show the status of a very,
 > very outdated version of Guix rather than the latest and greatest.
 > 
 > (Note that a very small number of the packages listed there are released
 > under restrictive “academic only” licenses or have undeclared licenses;
 > the license field says “non-free” or “undeclared”.  Having them on this
 > page does not mean I endorse the use of these tools and users should be
 > careful to check the license.)
 > 
 > > I have some notes contrasting GNU Guix with other similar/related
 > > tool-sets (modules, lmod, homebrew, spack, easyBuild, rolling rpms),
 > > as well as our current practices (sort of hybrid of rpms, bastard son
 > > of homebrew, and "just wedge it in there").  However I am not now
 > > intent upon setting up such a contrast, rather, I hope to focus on the
 > > advantages of GNU Guix in general.
 > 
 > I’d also like to point you at https://hal.inria.fr/hal-01161771/en, in
 > case you haven’t read it yet.  It includes a very general comparison
 > with tools like environment modules, spack, and easyBuild with a focus
 > on reproducibility.

Excellent - thanks - nice write-up - I'd somehow missed that earlier.
 
 > > + Guix detects and prohibits program name collisions; loading
 > >   conflicting packages into a users environment is _impossible_.  For
 > >   example, this prevents the ambiguity and associated error that can
 > >   arise when two packages define different programs by the same name
 > >   and both are placed in the user's execution PATH.
 > 
 > Well... I don’t know if this is worth pointing out.  Guix does detect
 > collisions but it doesn’t do anything intelligent here, maybe because
 > there isn’t much that can be done when a collision happens.  You mention
 > support for multiple profiles later, and I think that maybe this could
 > be merged.  I found collision detection to be not so useful because it
 > happens *during* profile generation, not *before* I commit to installing
 > a package.
 > > + Guix packages naturally extend to include all package resources,
 > >   including man pages, libraries, as well as binaries.  This is a
 > >   result of their using the underlying build engine's (i.e. gnu make)
 > >   installation targets.
 > 
 > Is there any package manager that does not handle this?

Alas, I have inherited a homebrew variant that is incapacitated in this regard.... but, enough of that!

> > + Guix packages, being independent of host on which they are built,
 > >   can be downloaded already built by upstream servers known as
 > >
 > [[https://www.gnu.org/software/guix/manual/html_node/Substitutes.html#Su
 > bstitutes][substitues]], with the assurance of their being bit-level identical
 > >   to the results of the generally longer process of configuration and
 > >   compilation on local servers.
 > 
 > Building packages in isolated environments is a necessary requirement
 > for bit-reproducibility, but it is not sufficient in itself.  To speak
 > of “assurance” is maybe a bit strong.
 > 
 > > + Guix packages, though dependent upon the machine architecture, are
 > >   independent on the linux distribution, or its version.  Thus, for
 > >   example, packages built under CentOS 6.5 on x86_64 will run under
 > >   operating system on x86_64, for example, CentOS 7.x and or Ubuntu
 > >   15.10 or 14.04.
 > 
 > This is true and it’s a great feature.  We’re using the very same Guix
 > store on various subversions of CentOS 6.x, a wide range of Ubuntus, and
 > Fedora.  “Linux distribution” is confusing, though.  I’d write
 > “GNU/Linux distribution”.  There *are* some kernel requirements, so the
 > version of Linux itself (i.e. the kernel version) does matter up to a
 > point for some of the advanced features of Guix (such as container
 > support).
 > 
 > > + Guix allows for unprivileged package management; users do not need
 > >   special elevated privileges (root or sudo) to create custom
 > >   environment profiles.  Especially noteworthy is that this allows
 > >   for rationally sharing package management as a distributed
 > >   responsibility.  This includes installing new site-available
 > >   applications (for those available in the guix repositories).  Thus,
 > >   if one user observes that a new release of a site-installed
 > >   application has become available, that user can safely install the
 > >   upgrade centrally and immediately start using it, without effecting
 >             ^—— what does this mean?  At the centre of ... what?
 > 
 > >   any other user's environment; without any further coordination,
 > >   when another user _is_ ready to adopt the upgrade, they will find
 > >   that installation is unnecessary, as it has already occurred.
 > 
 > I would clearly separate building and installing.  Installation is the
 > act of creating a new profile generation that contains a link to a
 > previously built (or substituted) item in the store.  Building (or
 > substituting) is what happens only once because of the purely functional
 > properties of the Guix package “functions”.
 > 
 > > + Guix provides a set of
 > [[https://www.gnu.org/software/guix/manual/guix.html#Build-Systems][build
 > systems]] providing support for language
 > >   specific package management systems, including R, perl, python,
 > >   ruby, haskell, and emacs.  This should allow a single approach for
 > >   managing computing environments for each of these languages/tools,
 > >   as opposed to needing to master the ideosyncracies of multiple
 > >   approaches, i.e. perlbrew for perl, pyenv or pythonbrew for python,
 > >   rbenv for ruby, etc.
 > 
 > Build systems are unrelated to “guix environment” or “rbenv”,
 > “virtualenv” and the like.  A build system is just a generalisation of
 > the steps that need to performed to build something.  For GNU-style
 > packages that’s
 > 
 >     ./configure --prefix=/something && make && make install
 > 
 > for Python it’s something like
 > 
 >     python setup.py install
 > 
> etc.  The build systems make *packaging* easier as we no longer need to
 > express the build procedure for R packages in terms of the GNU build
 > system (which would be inappropriate).  

Yes.  Agreed.  However, my emphasis here was intended to be that Guix can be used to obviate the need for rbenv, virtualenv, and friends.    I thought that `guix environment` was going to be an effective replacement for them.  Am I mistaken in this?   I hope not!  Assuming not, and if I understand your point, then I should write instead that this by virtue of guix's ability to set-up and tear down environments/profiles that not only specify versions of applications, but also libraries/plug-ins/modules for a variety of languages (ruby, perl, etc) and tools (emacs, etc).  You mention the importance of 'importers' below... perhaps it is the combination of available importers (for scaffolding the packaging from external repos) along with the ability to use `guix environment` to make them available in specified contexts.

Let me try.... Does this do better justice to characterizing the separation of concerns and attendant advantages:

 + Guix abstracts the idea of [[https://www.gnu.org/software/guix/manual/guix.html#Build-Systems][build systems]] to not only encompass the
   standard Gnu recipe "(configure; make; make check; make install)"
   but to also extend to common language/tool specific
   package/module/add-on managers, including those for R, perl,
   python, ruby, haskell, and emacs.  This should allow a single
   command line protocol for managing the building and deployment of
   such packages.  Additionally, when deployed using guix, such packages can
   also be enabled with guix using the `[[https://www.gnu.org/software/guix/manual/guix.html#Invoking-guix-environment][environment]]` sub-command,
   providing a unified approach where previously multiple tools might
   have been employed (perlbrew for perl, pyenv or pythonbrew for
   python, rbenv for ruby, etc.)


> This gives us the power of
 > abstraction, something that package managers like “conda” do not aim to
 > provide — there you need to ship a script along with the metadata file,
 > which is then run to build the package.  The script may need to do
 > little more than
 > 
 >     ./configure && make && make install
 > 
 > but it is a much less principled approach as bash has hardly the means
 > to provide for clear and easy abstraction.  (And to have to run
 > arbitrary shell scripts without any sort of isolation is not very
 > encouraging.)
 > 
 > The fact that build processes run as unprivileged, dedicated build users
 > is another feature: you don’t need to be afraid that the build script
 > breaks your precious home directory.  (See
 > https://github.com/MrMEEE/bumblebee-Old-and-
 > abbandoned/commit/a047be85247755cdbe0acce6f1dafc8beb84f2ac
 > for a shell script bug that isn’t all that rare and could really spoil
 > your day if you’re running a script like that without isolation.)

Heh - /usr rhymes with /lusr !

> > + Guix control of build processes down to level of
 > >   binary-compatibility, along with the management tools for
 > >   environment profiles can provide the basis for improvements in
 > >   reproducible pipelines, such as are begining to appear in
 > >   computational centers of excellence.
 > 
 > This sounds a bit confused.  When applying functional package management
 > ideas to a package and *all* of its dependencies recursively, you end up
 > with a directed acyclic graph (loops are “unrolled” by bootstrapping
 > with existing packages where possible) that is rooted in a very small
 > number of essential bootstrap binaries.  Together with build-time
 > isolation we get very close to a reproducible software stack.  The key
 > idea here, though, is functional package management because without it
 > build-time isolation wouldn’t be of much use.
 > 

Thanks.  Noted.

 > > + Guix provides commands to
 > >   [[https://www.gnu.org/software/guix/manual/guix.html#Emacs-
 > Commands][display
 > >   package information]] which can be used to automate the production
 > >   and publishing of software package catalogs.  Such catalogs may then
 > >   be shared, printed, emailed, or embedded into community-visible web
 > >   pages as various means of advertising package availability to the
 > >   research community.
 > 
 > Guix is a library, which allows for the Emacs interface and the Guix web
 > interface (which the MDC uses to display what software we have available
 > in Guix).
 > 
 > > + Guix package specifications, being written in
 > [[https://www.gnu.org/software/guix/manual/guix.html#Defining-
 > Packages][Guile/scheme]], do not
 > >   depend upon the users SHELL (i.e. guix works equally well with zsh,
 > >   bash, tcsh, etc) (TODO: confirm).
 > 
 > The user’s shell does not matter, but this is nothing to do with being
 > written in Scheme.  The Guix daemon spawns a shell on its own that does
 > not depend on the user’s environment.


What is not clear to me is whether bash is assumed elsewhere in guix.... for instance, the fact that 

	guix package --search-paths

reports environment variables in bash syntax.  Any insights here?


> > + Guix has a future - it is the package manager for the new
 > >   GNU-backed linux distribution,
 > [[https://www.gnu.org/software/guix/][GuixSD]].
 > 
 > “linux distribution” ...  GuixSD provides a variant of the GNU system
 > (which by default happens to use Linux as its kernel).

I will try and bear this distinction in mind.  

 > 
 > > + Guix is [[https://www.gnu.org/software/guix/manual/][well documented]].
 > 
 > Note that the manual on the website is somewhat outdated.  It’s the
 > manual for the latest 0.9.0 release (which is already quite some time
 > ago).  The latest version is only available in the git repository.
 > Maybe we should change this.
 > 
 >  > + Guix community has already prepared
 > [[https://www.gnu.org/software/guix/packages/][many recipes]], of which
 >  >   currently [[http://guix.mdc-berlin.de/packages?/?search=bioinfo][54 are
 > bioinformatics]] packages.
 > 
 > As I wrote above: it’s 114 (thanks for prompting me to fix the
 > configuration error that made it show 54 packages only).
 > 
 > > + Guix packaging is relatively easy to learn. It is reasonably
 > >   documented and there are `lint` style tools that check recipes for
 > >   being well-structured; they identify common errors in package
 > >   specification.
 > 
 > We also have importers (some with great, others with good results) so
 > that often there isn’t much work to be done at all.  We have importers
 > for CRAN and bioconductor, which worked pretty well for me; also the
 > hackage importer is great and it saved me a lot of time when I packaged
 > all missing dependencies for pandoc in about an afternoon.

Looking forward to that - getting pandoc and tex and Haskell and everything else lined up was a bear for me.

 > 
 > The importers are really very useful and I’d definitely mention them.
 > 
 > > + Guix development is open source.  It is open to input from all
 > >   community members.
 > 
 > It’s free software ;)
 > 
 > Maybe it’s also good to mention that you do not need to rely on Guix
 > upstream to add packages.  It is trivial to use custom packages with
 > GUIX_PACKAGE_PATH.
 > 
> > + Guix exposes the actual system calls to the package developer.
 > 
 > What does this mean?

Never mind.  Deleted.

> ~~ Ricardo

~~~Malcolm

Re: Making the case for GNU Guix ... advice sought

[Pjotr Prins]

On Thu, Feb 18, 2016 at 11:29:49PM +0000, Cook, Malcolm wrote:

> Yes.  Agreed.  However, my emphasis here was intended to be that
> Guix can be used to obviate the need for rbenv, virtualenv, and
> friends.    I thought that `guix environment` was going to be an
> effective replacement for them.  Am I mistaken in this?   

I have dropped rbenv, virtualenv and even bundler from my working
environments, thanks to Guix! I am really, really, really happy 
about that.

I even have different profiles for different ruby versions (one is on
1.8.7).

> I hope
> not!  Assuming not, and if I understand your point, then I should
> write instead that this by virtue of guix's ability to set-up and
> tear down environments/profiles that not only specify versions of
> applications, but also libraries/plug-ins/modules for a variety of
> languages (ruby, perl, etc) and tools (emacs, etc).  You mention the
> importance of 'importers' below... perhaps it is the combination of
> available importers (for scaffolding the packaging from external
> repos) along with the ability to use `guix environment` to make them
> available in specified contexts.

Yes, you need to create packages for all gems and Python modules in
use. Importers help define packages quickly.

Pj.

RE: Making the case for GNU Guix ... advice sought

[Cook, Malcolm]

Pjotr!

> > Yes.  Agreed.  However, my emphasis here was intended to be that
 > > Guix can be used to obviate the need for rbenv, virtualenv, and
 > > friends.    I thought that `guix environment` was going to be an
 > > effective replacement for them.  Am I mistaken in this?
 > 
 > I have dropped rbenv, virtualenv and even bundler from my working
 > environments, thanks to Guix! I am really, really, really happy
 > about that.
 > 
 > I even have different profiles for different ruby versions (one is on
 > 1.8.7).

Glad to hear to the positive news.  Looking forward to this too.

> > I hope
 > > not!  Assuming not, and if I understand your point, then I should
 > > write instead that this by virtue of guix's ability to set-up and
 > > tear down environments/profiles that not only specify versions of
 > > applications, but also libraries/plug-ins/modules for a variety of
 > > languages (ruby, perl, etc) and tools (emacs, etc).  You mention the
 > > importance of 'importers' below... perhaps it is the combination of
 > > available importers (for scaffolding the packaging from external
 > > repos) along with the ability to use `guix environment` to make them
 > > available in specified contexts.
 > 
 > Yes, you need to create packages for all gems and Python modules in
 > use. Importers help define packages quickly.

Again, thanks for confirmation, and for your succinct paraphrasing of this consideration.

~Malcolm

 > 
 > Pj.

Re: Making the case for GNU Guix ... advice sought

[Ludovic Courtès]

"Cook, Malcolm" <MEC@stowers.org> skribis:

> What is not clear to me is whether bash is assumed elsewhere in guix.... for instance, the fact that 
>
> 	guix package --search-paths
>
> reports environment variables in bash syntax.

In practice it’s almost plain Bourne shell syntax.  I think the only
“Bashism” is the mixed assignment and export (“export VAR=value”), which
is supported by most Bourne-compatible shells I believe (Dash being an
exception.)

> Any insights here?

The ‘etc/profile’ file that is created within profiles, such as
~/.guix-profile/etc/profile, uses the same syntax as above.

I don’t think there are other places in the user interfaces where Bash
is expected.

Thanks,
Ludo’.

Re: Making the case for GNU Guix ... advice sought

[Ricardo Wurmus]

Cook, Malcolm <MEC@stowers.org> writes:

> Hi Ricardo,
>
> Great specifics.  If you can help me solve a few of my remaining
> questions, in-line below, I'd be much obliged....

Hi Malcolm,

I see that others have already replied to some of your questions.  Are
there any other things that you’d like us to clarify?

~~ Ricardo

RE: Making the case for GNU Guix ... advice sought

[Cook, Malcolm]

> Cook, Malcolm <MEC@stowers.org> writes:
 > 
 > > Hi Ricardo,
 > >
 > > Great specifics.  If you can help me solve a few of my remaining
 > > questions, in-line below, I'd be much obliged....
 > 
 > Hi Malcolm,
 > 
 > I see that others have already replied to some of your questions.  Are
 > there any other things that you’d like us to clarify?
[Cook, Malcolm] 

No - but thanks for reaching out - I'm sure I will have more questions as I slowly advance.

Insofar as my "MakingTheCase" document, I have finished it to my current satisfaction and have used it internally to communicate my perceived value of moving forward with GUIX.

https://github.com/malcook/sce/blob/master/MakingTheCase.org

Again, thank everyone for your helps.  If the Guix project wanted to incorporate this document with whatever improvements, I'd be happy to contribute.... just say the word....

Cheers,

Malcolm


 > 
 > ~~ Ricardo

RE: Making the case for GNU Guix ... advice sought

[Cook, Malcolm]

By the way, below my sig is my new and improved "wrap" to my "pitch".  I hope you like it...  
~Malcolm

Guix provides a well-engineered, supportable, portable, and extensible approach to software package and environment management. It allows to adopt and create unambiguous, clearly-specified, stand-alone, and reproducible software environments. It provides the administrative flexibility to deploy with transactional control a common standard suite of applications, as well as the agility to quickly respond to individual user and application requirements for more customized environments. Guix reduces the administrative burden for software package and environment management by allowing it to be distributed among researchers, with a natural means of coordination of effort. By offering a unified approach to environment management, it replaces the need for multiple other such tools, reducing support overhead. Guix can serve the needs for computational reproducibility through means of sharing these environments across a range of hosts and computer operating systems with both internal collaborators as well as the broader research community.

In short, Guix provides a rigorous response to requirements for package and environment integrity in a rapidly evolving research computing environment while easing and distributing the burden of administration. 


 > -----Original Message-----
 > From: guix-devel-bounces+mec=stowers.org@gnu.org [mailto:guix-devel-
 > bounces+mec=stowers.org@gnu.org] On Behalf Of Cook, Malcolm
 > Sent: Monday, February 29, 2016 10:55 AM
 > To: 'Ricardo Wurmus' <ricardo.wurmus@mdc-berlin.de>
 > Cc: guix-devel@gnu.org; 'bio-packaging@mailman.open-bio.org' <bio-
 > packaging@mailman.open-bio.org>
 > Subject: RE: Making the case for GNU Guix ... advice sought
 > 
 > > Cook, Malcolm <MEC@stowers.org> writes:
 >  >
 >  > > Hi Ricardo,
 >  > >
 >  > > Great specifics.  If you can help me solve a few of my remaining
 >  > > questions, in-line below, I'd be much obliged....
 >  >
 >  > Hi Malcolm,
 >  >
 >  > I see that others have already replied to some of your questions.  Are
 >  > there any other things that you’d like us to clarify?
 > [Cook, Malcolm]
 > 
 > No - but thanks for reaching out - I'm sure I will have more questions as I
 > slowly advance.
 > 
 > Insofar as my "MakingTheCase" document, I have finished it to my current
 > satisfaction and have used it internally to communicate my perceived value of
 > moving forward with GUIX.
 > 
 > https://github.com/malcook/sce/blob/master/MakingTheCase.org
 > 
 > Again, thank everyone for your helps.  If the Guix project wanted to
 > incorporate this document with whatever improvements, I'd be happy to
 > contribute.... just say the word....
 > 
 > Cheers,
 > 
 > Malcolm
 > 
 > 
 >  >
 >  > ~~ Ricardo

Re: Making the case for GNU Guix ... advice sought

[Ludovic Courtès]

"Cook, Malcolm" <MEC@stowers.org> skribis:

> By the way, below my sig is my new and improved "wrap" to my "pitch".  I hope you like it...  

It’s quite strong, but I like it.  Thanks!  :-)

Ludo’.