From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ricardo Wurmus Subject: Re: Making the case for GNU Guix ... advice sought Date: Tue, 9 Feb 2016 16:03:13 +0100 Message-ID: References: <953f16876af34ea59b526b7f35482e72@exchsrv2.sgc.loc> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:43411) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aT9pC-00037b-Uh for guix-devel@gnu.org; Tue, 09 Feb 2016 10:03:37 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aT9p7-0006zz-M9 for guix-devel@gnu.org; Tue, 09 Feb 2016 10:03:30 -0500 Received: from pegasus.bbbm.mdc-berlin.de ([141.80.25.20]:39337) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aT9p7-0006z8-6A for guix-devel@gnu.org; Tue, 09 Feb 2016 10:03:25 -0500 In-Reply-To: <953f16876af34ea59b526b7f35482e72@exchsrv2.sgc.loc> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: "Cook, Malcolm" Cc: guix-devel@gnu.org, "'bio-packaging@mailman.open-bio.org'" Cook, Malcolm writes: > I have been asked to write up an argument for the advantages that GNU > Guix confers for deploying linux software, especially scientific > computing and including bioinformatics software. > =09 > To that end I have written up this document=20 > > https://github.com/malcook/sce/blob/master/MakingTheCase.org Nice! Here=E2=80=99s a first correction for a mistake that I=E2=80=99m responsi= ble for: the number of bioinfo packages is actually closer to 114 (not 54). Guix web was misconfigured on that host and it would show the status of a very, very outdated version of Guix rather than the latest and greatest. (Note that a very small number of the packages listed there are released under restrictive =E2=80=9Cacademic only=E2=80=9D licenses or have undecl= ared licenses; the license field says =E2=80=9Cnon-free=E2=80=9D or =E2=80=9Cundeclared=E2= =80=9D. Having them on this page does not mean I endorse the use of these tools and users should be careful to check the license.) > I have some notes contrasting GNU Guix with other similar/related > tool-sets (modules, lmod, homebrew, spack, easyBuild, rolling rpms), > as well as our current practices (sort of hybrid of rpms, bastard son > of homebrew, and "just wedge it in there"). However I am not now > intent upon setting up such a contrast, rather, I hope to focus on the > advantages of GNU Guix in general. I=E2=80=99d also like to point you at https://hal.inria.fr/hal-01161771/e= n, in case you haven=E2=80=99t read it yet. It includes a very general compari= son with tools like environment modules, spack, and easyBuild with a focus on reproducibility. > + Guix detects and prohibits program name collisions; loading > conflicting packages into a users environment is _impossible_. For > example, this prevents the ambiguity and associated error that can > arise when two packages define different programs by the same name > and both are placed in the user's execution PATH. Well... I don=E2=80=99t know if this is worth pointing out. Guix does de= tect collisions but it doesn=E2=80=99t do anything intelligent here, maybe bec= ause there isn=E2=80=99t much that can be done when a collision happens. You = mention support for multiple profiles later, and I think that maybe this could be merged. I found collision detection to be not so useful because it happens *during* profile generation, not *before* I commit to installing a package. > + Guix packages naturally extend to include all package resources, > including man pages, libraries, as well as binaries. This is a > result of their using the underlying build engine's (i.e. gnu make) > installation targets. Is there any package manager that does not handle this? > + Guix packages, being independent of host on which they are built, > can be downloaded already built by upstream servers known as > [[https://www.gnu.org/software/guix/manual/html_node/Substitutes.html= #Substitutes][substitues]], with the assurance of their being bit-level i= dentical > to the results of the generally longer process of configuration and > compilation on local servers. Building packages in isolated environments is a necessary requirement for bit-reproducibility, but it is not sufficient in itself. To speak of =E2=80=9Cassurance=E2=80=9D is maybe a bit strong. > + Guix packages, though dependent upon the machine architecture, are > independent on the linux distribution, or its version. Thus, for > example, packages built under CentOS 6.5 on x86_64 will run under > operating system on x86_64, for example, CentOS 7.x and or Ubuntu > 15.10 or 14.04. This is true and it=E2=80=99s a great feature. We=E2=80=99re using the v= ery same Guix store on various subversions of CentOS 6.x, a wide range of Ubuntus, and Fedora. =E2=80=9CLinux distribution=E2=80=9D is confusing, though. I=E2= =80=99d write =E2=80=9CGNU/Linux distribution=E2=80=9D. There *are* some kernel requir= ements, so the version of Linux itself (i.e. the kernel version) does matter up to a point for some of the advanced features of Guix (such as container support). > + Guix allows for unprivileged package management; users do not need > special elevated privileges (root or sudo) to create custom > environment profiles. Especially noteworthy is that this allows > for rationally sharing package management as a distributed > responsibility. This includes installing new site-available > applications (for those available in the guix repositories). Thus, > if one user observes that a new release of a site-installed > application has become available, that user can safely install the > upgrade centrally and immediately start using it, without effecting ^=E2=80=94=E2=80=94 what does this mean? At the centre of ..= . what? > any other user's environment; without any further coordination, > when another user _is_ ready to adopt the upgrade, they will find > that installation is unnecessary, as it has already occurred. I would clearly separate building and installing. Installation is the act of creating a new profile generation that contains a link to a previously built (or substituted) item in the store. Building (or substituting) is what happens only once because of the purely functional properties of the Guix package =E2=80=9Cfunctions=E2=80=9D. > + Guix provides a set of [[https://www.gnu.org/software/guix/manual/gui= x.html#Build-Systems][build systems]] providing support for language > specific package management systems, including R, perl, python, > ruby, haskell, and emacs. This should allow a single approach for > managing computing environments for each of these languages/tools, > as opposed to needing to master the ideosyncracies of multiple > approaches, i.e. perlbrew for perl, pyenv or pythonbrew for python, > rbenv for ruby, etc. Build systems are unrelated to =E2=80=9Cguix environment=E2=80=9D or =E2=80= =9Crbenv=E2=80=9D, =E2=80=9Cvirtualenv=E2=80=9D and the like. A build system is just a gene= ralisation of the steps that need to performed to build something. For GNU-style packages that=E2=80=99s ./configure --prefix=3D/something && make && make install for Python it=E2=80=99s something like python setup.py install etc. The build systems make *packaging* easier as we no longer need to express the build procedure for R packages in terms of the GNU build system (which would be inappropriate). This gives us the power of abstraction, something that package managers like =E2=80=9Cconda=E2=80=9D= do not aim to provide =E2=80=94 there you need to ship a script along with the metadata= file, which is then run to build the package. The script may need to do little more than ./configure && make && make install but it is a much less principled approach as bash has hardly the means to provide for clear and easy abstraction. (And to have to run arbitrary shell scripts without any sort of isolation is not very encouraging.) The fact that build processes run as unprivileged, dedicated build users is another feature: you don=E2=80=99t need to be afraid that the build sc= ript breaks your precious home directory. (See https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/commit/a047be85247= 755cdbe0acce6f1dafc8beb84f2ac for a shell script bug that isn=E2=80=99t all that rare and could really = spoil your day if you=E2=80=99re running a script like that without isolation.) > + Guix control of build processes down to level of > binary-compatibility, along with the management tools for > environment profiles can provide the basis for improvements in > reproducible pipelines, such as are begining to appear in > computational centers of excellence. This sounds a bit confused. When applying functional package management ideas to a package and *all* of its dependencies recursively, you end up with a directed acyclic graph (loops are =E2=80=9Cunrolled=E2=80=9D by bo= otstrapping with existing packages where possible) that is rooted in a very small number of essential bootstrap binaries. Together with build-time isolation we get very close to a reproducible software stack. The key idea here, though, is functional package management because without it build-time isolation wouldn=E2=80=99t be of much use. > + Guix provides commands to > [[https://www.gnu.org/software/guix/manual/guix.html#Emacs-Commands][= display > package information]] which can be used to automate the production > and publishing of software package catalogs. Such catalogs may then > be shared, printed, emailed, or embedded into community-visible web > pages as various means of advertising package availability to the > research community. Guix is a library, which allows for the Emacs interface and the Guix web interface (which the MDC uses to display what software we have available in Guix). > + Guix package specifications, being written in [[https://www.gnu.org/s= oftware/guix/manual/guix.html#Defining-Packages][Guile/scheme]], do not > depend upon the users SHELL (i.e. guix works equally well with zsh, > bash, tcsh, etc) (TODO: confirm). The user=E2=80=99s shell does not matter, but this is nothing to do with = being written in Scheme. The Guix daemon spawns a shell on its own that does not depend on the user=E2=80=99s environment. > + Guix has a future - it is the package manager for the new > GNU-backed linux distribution, [[https://www.gnu.org/software/guix/][= GuixSD]]. =E2=80=9Clinux distribution=E2=80=9D ... GuixSD provides a variant of th= e GNU system (which by default happens to use Linux as its kernel). > + Guix is [[https://www.gnu.org/software/guix/manual/][well documented]= ]. Note that the manual on the website is somewhat outdated. It=E2=80=99s t= he manual for the latest 0.9.0 release (which is already quite some time ago). The latest version is only available in the git repository. Maybe we should change this. > + Guix community has already prepared [[https://www.gnu.org/software/g= uix/packages/][many recipes]], of which > currently [[http://guix.mdc-berlin.de/packages?/?search=3Dbioinfo][5= 4 are bioinformatics]] packages. As I wrote above: it=E2=80=99s 114 (thanks for prompting me to fix the configuration error that made it show 54 packages only). > + Guix packaging is relatively easy to learn. It is reasonably > documented and there are `lint` style tools that check recipes for > being well-structured; they identify common errors in package > specification. We also have importers (some with great, others with good results) so that often there isn=E2=80=99t much work to be done at all. We have impo= rters for CRAN and bioconductor, which worked pretty well for me; also the hackage importer is great and it saved me a lot of time when I packaged all missing dependencies for pandoc in about an afternoon. The importers are really very useful and I=E2=80=99d definitely mention t= hem. > + Guix development is open source. It is open to input from all > community members. It=E2=80=99s free software ;) Maybe it=E2=80=99s also good to mention that you do not need to rely on G= uix upstream to add packages. It is trivial to use custom packages with GUIX_PACKAGE_PATH. > + Guix exposes the actual system calls to the package developer. What does this mean? ~~ Ricardo