From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Cook, Malcolm" Subject: Using a shared Guix store (was RE: [Bio-packaging] testing out guix) Date: Thu, 18 Jun 2015 20:22:24 +0000 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:44348) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z5gKV-0002YG-KF for guix-devel@gnu.org; Thu, 18 Jun 2015 16:22:32 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Z5gKR-0007s4-Cy for guix-devel@gnu.org; Thu, 18 Jun 2015 16:22:31 -0400 Received: from smtp01.stowers.org ([40.141.174.61]:55851) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z5gKR-0007rR-6a for guix-devel@gnu.org; Thu, 18 Jun 2015 16:22:27 -0400 Content-Language: en-US List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: 'Ricardo Wurmus' , 'Pjotr Prins' Cc: Guix-devel , "'bio-packaging@mailman.open-bio.org'" Ricardo, Pj, et al, ... Hi - I am resending this message as I omitted Guix-devel at first and would like to cast a broader net.... I am interested in understanding details behind Ricardo's observation: "Gui= x is not designed to be run in a centralised manner. A Guix daemon is suppo= sed to run on each system as root and it listens to RPCs from local users o= nly. In an environment with multiple clusters and multiple workstations thi= s approach requires considerable effort to make it work correctly and secur= ely. Instead we opted to run the Guix daemon on a single dedicated server, = writing profile data and store items onto an NFS share. . The cluster nodes= and workstations mount this share read-only." (http://elephly.net/posts/20= 15-04-17-gnu-guix.html) Can anyone elaborate a little on what are the obstacles to having `/gnu` m= ounted read-write network wide? Can it be partially characterized as: Multi-process contention over un-coordinated access to the store (especial= ly write access necessitated by supporting the `build` action) If so, might this be mitigated using a variant off of "Using the Offload Fa= cility" (http://www.gnu.org/software/guix/manual/guix.html#Daemon-Offload-S= etup) in which builds would still be offloaded (and thus subject to coordin= ation), with the elimination of the need for " Missing prerequisites for th= e build are copied over SSH to the target machine, which then proceeds with= the build; upon success the output(s) of the build are copied back to the = initial machine" since they would be done through the shared file system? Do I understand correctly that in your setup, Ricardo, that absolutely no `= guix` commands are executed on any host other than the "single dedicated se= rver". What about `guix environment p1 p2 p3` when p1 p2 p3 are already a= vailable in /gnu. If I understand correctly, in such a case, nothing need = be written to /gnu... and so should not present any challenge to running gu= ix off a shared mount. Or am I missing an aspect of what is going on? Lacking the ability to, from any host, dynamically alter the runtime enviro= nment to include _already installed_ scientific applications seems like th= e most important aspect of guix that would be untolerable to my users were = I to user guix. I do hope I am mis-understanding the trade-off you made = at your site. I think the second-worst trade-off this compromise takes is that a user can= not even alter their own profile unless connected to the master guix host. = For instance, a user switching her default emacs between two already buil= t & installed versions only requires editing the profiles/per-user symlink = farm; couldn't such commands be put behind a per-user semaphore, allowing = guix/profiles/per-user to be made available +rw on a shared network mount? I really appreciate everyone's assistance in helping me understand these tr= ade-offs, how guix works, and if there is any new or different thinking abo= ut strategies for deploying guix. Thanks, Malcolm Cook Hi Malcolm, > Pleased to e-meet you. It was your > http://elephly.net/posts/2015-04-17-gnu-guix.html that got on this=20 > path. I'm so glad I found it. Oh, great to read that it reached someone who might benefit from it :) > Great. I have a trove of recipes (in either bash and a > brew-derivative) which I intend to move into guix over time, unless=20 > I'm beaten to any of them (I'm sure I already have been for some):> [...] I just went through the list and found a few that looked familiar. Below are some comments. Non-free, won't package: > gatk > jimKentUtils On my list (possibly non-free): > igv > igvtools > meme > picard (requires more Java toolchain stuff to build from source) > soapdenovo2 > trimmomatic Non-free stuff, packaged in my private guix-nonfree repo: > macs14 > tophat > viennaRNA Free stuff, packaged and available through Guix: > bamtools > bedtools > blast+ (WIP, it's really big) > bowtie2 > cufflinks > fastqc > fastx_toolkit > graphviz > ncbi_sra_toolkit > ngsutils > rsem > star > tabix These look familiar but I'm not sure if I already have them: > bam2fastq > bcftools [...] > Gonna be a party! Indeed! I would be very happy if you decide for Guix in the end. There is= n't a lot of bioinfo stuff there yet, but that can be fixed by having more = people contribute recipes. Cheers, Ricardo