Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Cook, Malcolm]

Ricardo, Pj, et al,


... Hi - I am resending this message as I omitted Guix-devel <guix-devel@gnu.org> at first and would like to cast a broader net....


I am interested in understanding details behind Ricardo's observation: "Guix is not designed to be run in a centralised manner. A Guix daemon is supposed to run on each system as root and it listens to RPCs from local users only. In an environment with multiple clusters and multiple workstations this approach requires considerable effort to make it work correctly and securely. Instead we opted to run the Guix daemon on a single dedicated server, writing profile data and store items onto an NFS share. . The cluster nodes and workstations mount this share read-only." (http://elephly.net/posts/2015-04-17-gnu-guix.html)

Can anyone elaborate a little on what are the obstacles to having  `/gnu` mounted read-write network wide?

Can it be partially characterized as:

	Multi-process contention over un-coordinated access to the store (especially write access necessitated by supporting the `build` action)

If so, might this be mitigated using a variant off of "Using the Offload Facility" (http://www.gnu.org/software/guix/manual/guix.html#Daemon-Offload-Setup) in which builds would still be offloaded (and thus subject to coordination), with the elimination of the need for " Missing prerequisites for the build are copied over SSH to the target machine, which then proceeds with the build; upon success the output(s) of the build are copied back to the initial machine" since they would be done through the shared file system?

Do I understand correctly that in your setup, Ricardo, that absolutely no `guix` commands are executed on any host other than the "single dedicated server".   What about `guix environment p1 p2 p3` when p1 p2 p3 are already available in /gnu.  If I understand correctly, in such a case, nothing need be written to /gnu... and so should not present any challenge to running guix off a shared mount.   Or am I missing an aspect of what is going on?

Lacking the ability to, from any host, dynamically alter the runtime environment to include _already installed_  scientific applications seems like the most important aspect of guix that would be untolerable to my users were I to user guix.    I do hope I am mis-understanding the trade-off you made at your site.

I think the second-worst trade-off this compromise takes is that a user cannot even alter their own profile unless connected to the master guix host.  For instance, a user switching her default emacs  between two already built & installed versions only requires editing the profiles/per-user symlink farm; couldn't such commands be put behind a per-user semaphore,  allowing guix/profiles/per-user to be made available +rw on a shared network mount?

I really appreciate everyone's assistance in helping me understand these trade-offs, how guix works, and if there is any new or different thinking about strategies for deploying guix.

Thanks,

Malcolm Cook



Hi Malcolm,

> Pleased to e-meet you.  It was your
> http://elephly.net/posts/2015-04-17-gnu-guix.html that got on this 
> path.  I'm so glad I found it.

Oh, great to read that it reached someone who might benefit from it :)

> Great.  I have a trove of recipes (in either bash and a
> brew-derivative) which I intend to move into guix over time, unless 
> I'm beaten to any of them (I'm sure I already have been for some):>

[...]

I just went through the list and found a few that looked familiar.
Below are some comments.

Non-free, won't package:

> gatk
> jimKentUtils

On my list (possibly non-free):

> igv
> igvtools
> meme
> picard (requires more Java toolchain stuff to build from source)
> soapdenovo2
> trimmomatic

Non-free stuff, packaged in my private guix-nonfree repo:

> macs14
> tophat
> viennaRNA

Free stuff, packaged and available through Guix:

> bamtools
> bedtools
> blast+  (WIP, it's really big)
> bowtie2
> cufflinks
> fastqc
> fastx_toolkit
> graphviz
> ncbi_sra_toolkit
> ngsutils
> rsem
> star
> tabix

These look familiar but I'm not sure if I already have them:

> bam2fastq
> bcftools

[...]

> Gonna be a party!

Indeed!  I would be very happy if you decide for Guix in the end.  There isn't a lot of bioinfo stuff there yet, but that can be fixed by having more people contribute recipes.

Cheers,
Ricardo

Re: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Ricardo Wurmus]

Cook, Malcolm <MEC@stowers.org> writes:

> Can anyone elaborate a little on what are the obstacles to having
> `/gnu` mounted read-write network wide?

Yes, the primary problem is that the daemon assumes that it is the only
thing writing to the store and the localstatedir.  Any modification of
profiles and the store goes through the daemon.

> If so, might this be mitigated using a variant off of "Using the
> Offload Facility"
> (http://www.gnu.org/software/guix/manual/guix.html#Daemon-Offload-Setup)
> in which builds would still be offloaded (and thus subject to
> coordination), with the elimination of the need for " Missing
> prerequisites for the build are copied over SSH to the target machine,
> which then proceeds with the build; upon success the output(s) of the
> build are copied back to the initial machine" since they would be done
> through the shared file system?

Something like that has been suggested before: if the daemon were to
accept authenticated connections from the outside rather than to just
listen on a local socket we could have remote guix clients connecting to
the central daemon.

> Do I understand correctly that in your setup, Ricardo, that absolutely
> no `guix` commands are executed on any host other than the "single
> dedicated server".  What about `guix environment p1 p2 p3` when p1 p2
> p3 are already available in /gnu.  If I understand correctly, in such
> a case, nothing need be written to /gnu... and so should not present
> any challenge to running guix off a shared mount.  Or am I missing an
> aspect of what is going on?

This is correct.  Our needs are rather simple, so people don’t use any
of the guix commands on their local machines, nor on the cluster nodes.
This is not as inconvenient as this may seem when profiles don’t change
often, but it is of course a serious limitation.  My first goal was to
replace traditional software *packaging* (because that’s what affected
my work as a sysadmin); the next goal is to allow users to conveniently
perform software *management*.  At the moment this requires users to log
on to the guix management host (or to ask a sysadmin to perform the
changes for them).

Currently our users are okay with that, probably to a large part because
they are not yet aware of all the features of Guix.  They are only used
to management by sysadmins or manual compilation, so they are not
inconvenienced.

Ultimately, the correct fix is to allow remote guix clients to
communicate with a central guix daemon.  The daemon does not even need
to be aware of remote connections if guix clients can transparently
connect via SSH and send RPCs to the socket.  This is not yet
implemented.

Even then, /gnu would be mounted read-only on all but the management
host, because there would still only be one daemon writing to the store
and the localstatedir – even if it would take requests from multiple
remote guix clients.

~~ Ricardo

Re: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Ludovic Courtès]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> Something like that has been suggested before: if the daemon were to
> accept authenticated connections from the outside rather than to just
> listen on a local socket we could have remote guix clients connecting to
> the central daemon.

[...]

> Ultimately, the correct fix is to allow remote guix clients to
> communicate with a central guix daemon.  The daemon does not even need
> to be aware of remote connections if guix clients can transparently
> connect via SSH and send RPCs to the socket.  This is not yet
> implemented.

This could be either SSH or a plain direct connection, which is
acceptable on a LAN.

A simple way to test the latter is with ‘socat’, something along the
lines of what I described at:

  https://lists.gnu.org/archive/html/guix-devel/2015-02/msg00680.html

It would be great if you could check whether something like this works
for you.

Then we could also have built-in support for that in the daemon and
clients, so that it’s more convenient.

Thanks,
Ludo’.

Re: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> A simple way to test the latter is with ‘socat’, something along the
> lines of what I described at:
>
>   https://lists.gnu.org/archive/html/guix-devel/2015-02/msg00680.html
>
> It would be great if you could check whether something like this works
> for you.
>
> Then we could also have built-in support for that in the daemon and
> clients, so that it’s more convenient.

This is still on my list.  Maybe I can get around to testing this today
or tomorrow.

~~ Ricardo

Re: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Ludovic Courtès]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> Even then, /gnu would be mounted read-only on all but the management
> host, because there would still only be one daemon writing to the store
> and the localstatedir – even if it would take requests from multiple
> remote guix clients.

I think $localstatedir (/var/guix by default), or a subset thereof
(namely /var/guix/profiles/per-user) should be shared read-write over
NFS to allow users to create the profiles/per-user/$USER symlinks from
other machines.

Sysadmins might find it more convenient to have $localstatedir under
/gnu, for instance --localstatedir=/gnu/var/guix, so that simply
exporting /gnu over NFS covers everything.

(Of course you still need the ability to talk to the central daemon from
any host, as we discussed.)

Thanks,
Ludo’.

RE: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Cook, Malcolm]

Hi Ludovic and Ricardo,

> Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:
> 
> > Even then, /gnu would be mounted read-only on all but the management
> > host, because there would still only be one daemon writing to the
> > store and the localstatedir – even if it would take requests from
> > multiple remote guix clients.
> 
> I think $localstatedir (/var/guix by default), or a subset thereof (namely
> /var/guix/profiles/per-user) should be shared read-write over NFS to allow
> users to create the profiles/per-user/$USER symlinks from other machines.

Loduvic, I'm not sure I understand well enough to agree....

Ricardo, I appreciate your fixing my reasoning below....

Certainly per-user profiles must be network readable , since they wind up on PATH.   Perhaps within /gnu/guix/profiles/per-user?

However, if `guix` command is enabled (using ssh/pipe/??) to run in networked environment with build and other work continuing to be coordinated by guix-daemon, then updates to per-user profiles similarly would need to be coordinated.  For example, if a user wants to `guix package --install  somePkg-someVersion` then that package must  be protected against any "simultaneous" garbage collection.  Running the --install through the network daemon is the guix way of protecting against such a race condition.

On this topic, examining the documentation a bit further makes me wonder whether support for `guix build --root  MYROOT` will be impossible when (a) MYROOT is not accessible to the host running guix-daemon.  Will this be an issue?  

Cheers,

Malcolm

Re: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Ricardo Wurmus]

Cook, Malcolm <MEC@stowers.org> writes:

> Certainly per-user profiles must be network readable , since they wind
> up on PATH.  Perhaps within /gnu/guix/profiles/per-user?

I have configured guix such that the $localstatedir is /gnu/var, so I
only have to export /gnu.

> However, if `guix` command is enabled (using ssh/pipe/??) to run in
> networked environment with build and other work continuing to be
> coordinated by guix-daemon, then updates to per-user profiles
> similarly would need to be coordinated.  For example, if a user wants
> to `guix package --install somePkg-someVersion` then that package must
> be protected against any "simultaneous" garbage collection.  Running
> the --install through the network daemon is the guix way of protecting
> against such a race condition.

Correct.  With a shared store there must be a shared daemon.  An
alternative setup would be to have completely separate stores and reduce
build work by using ‘guix publish’.

Unfortunately, I have not yet been able to test a setup where the local
‘guix’ command talks via SSH/socat to a central guix-daemon.  (I’ve been
trying to improve the performance in our situation where the store is
located on a central file server and the daemon operates on the NFS
share.  This is rather slow here and could only partially be improved by
disabling deduplication.)

~~ Ricardo

RE: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Cook, Malcolm]

Ricardo,

Thanks for your complete and easy-to-understand response to my inquiries.

> > Can anyone elaborate a little on what are the obstacles to having
> > `/gnu` mounted read-write network wide?
> 
> Yes, the primary problem is that the daemon assumes that it is the only thing
> writing to the store and the localstatedir.  Any modification of profiles and
> the store goes through the daemon.
> > If so, might this be mitigated using a variant off of "Using the
> > Offload Facility"
> > (http://www.gnu.org/software/guix/manual/guix.html#Daemon-Offload-
> Setu
> > p) in which builds would still be offloaded (and thus subject to
> > coordination), with the elimination of the need for " Missing
> > prerequisites for the build are copied over SSH to the target machine,
> > which then proceeds with the build; upon success the output(s) of the
> > build are copied back to the initial machine" since they would be done
> > through the shared file system?
> 
> Something like that has been suggested before: if the daemon were to
> accept authenticated connections from the outside rather than to just listen
> on a local socket we could have remote guix clients connecting to the central
> daemon.

Hmm.  So, for now, I could just teach users to, i.e., `ssh ${USER}@${GUIX_HOST}   /path/to/guix package -r lua -i guile guile-cairo` and contrive for GUIX_HOST to be defined for them.

Or, wrap it up even further, and contrive for everyone to have the following in their env:

	alias guixpkg='ssh ${USER}@${GUIX_HOST} /path/to/guix package'

??

> > Do I understand correctly that in your setup, Ricardo, that absolutely
> > no `guix` commands are executed on any host other than the "single
> > dedicated server".  What about `guix environment p1 p2 p3` when p1 p2
> > p3 are already available in /gnu.  If I understand correctly, in such
> > a case, nothing need be written to /gnu... and so should not present
> > any challenge to running guix off a shared mount.  Or am I missing an
> > aspect of what is going on?
> 
> This is correct.  Our needs are rather simple, so people don’t use any of the
> guix commands on their local machines, nor on the cluster nodes.
> This is not as inconvenient as this may seem when profiles don’t change
> often, but it is of course a serious limitation.  My first goal was to replace
> traditional software *packaging* (because that’s what affected my work as a
> sysadmin); the next goal is to allow users to conveniently perform software
> *management*.  At the moment this requires users to log on to the guix
> management host (or to ask a sysadmin to perform the changes for them).

I totally get your priorities.  Thanks for the reveal!  Are these also the priorities of the GnuSD project, do you know?

> Currently our users are okay with that, probably to a large part because they
> are not yet aware of all the features of Guix.  They are only used to
> management by sysadmins or manual compilation, so they are not
> inconvenienced.
> 
> Ultimately, the correct fix is to allow remote guix clients to communicate with
> a central guix daemon.  The daemon does not even need to be aware of
> remote connections if guix clients can transparently connect via SSH and send
> RPCs to the socket.  This is not yet implemented.

Sounds great.  On the roadmap?

> Even then, /gnu would be mounted read-only on all but the management
> host, because there would still only be one daemon writing to the store and
> the localstatedir – even if it would take requests from multiple remote guix
> clients.

Agreed.  

Fingers crossed, and thanks again,

~Malcolm


> 
> ~~ Ricardo

Re: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Ludovic Courtès]

Hi,

"Cook, Malcolm" <MEC@stowers.org> skribis:

>> > Can anyone elaborate a little on what are the obstacles to having
>> > `/gnu` mounted read-write network wide?
>> 
>> Yes, the primary problem is that the daemon assumes that it is the only thing
>> writing to the store and the localstatedir.  Any modification of profiles and
>> the store goes through the daemon.
>> > If so, might this be mitigated using a variant off of "Using the
>> > Offload Facility"
>> > (http://www.gnu.org/software/guix/manual/guix.html#Daemon-Offload-
>> Setu
>> > p) in which builds would still be offloaded (and thus subject to
>> > coordination), with the elimination of the need for " Missing
>> > prerequisites for the build are copied over SSH to the target machine,
>> > which then proceeds with the build; upon success the output(s) of the
>> > build are copied back to the initial machine" since they would be done
>> > through the shared file system?
>> 
>> Something like that has been suggested before: if the daemon were to
>> accept authenticated connections from the outside rather than to just listen
>> on a local socket we could have remote guix clients connecting to the central
>> daemon.
>
> Hmm.  So, for now, I could just teach users to, i.e., `ssh ${USER}@${GUIX_HOST}   /path/to/guix package -r lua -i guile guile-cairo` and contrive for GUIX_HOST to be defined for them.
>
> Or, wrap it up even further, and contrive for everyone to have the following in their env:
>
> 	alias guixpkg='ssh ${USER}@${GUIX_HOST} /path/to/guix package'
>
> ??

That’s a possibility.

Another one, which I think Ricardo has been investigating lately, would
be to have users manage their profiles via the Web user interface,
guix-web: <https://git.dthompson.us/guix-web.git>.  (The video
<http://audio-video.gnu.org/video/misc/2015-01__GNU_Guix__The_Emacs_of_Distros.webm>
has a demo starting at around 23mn.)

>> Currently our users are okay with that, probably to a large part because they
>> are not yet aware of all the features of Guix.  They are only used to
>> management by sysadmins or manual compilation, so they are not
>> inconvenienced.
>> 
>> Ultimately, the correct fix is to allow remote guix clients to communicate with
>> a central guix daemon.  The daemon does not even need to be aware of
>> remote connections if guix clients can transparently connect via SSH and send
>> RPCs to the socket.  This is not yet implemented.
>
> Sounds great.  On the roadmap?

Definitely.  There are details to be sorted out (SSH vs. plain old
socket), but we should discuss it and “get it done.”

Thanks for your feedback,
Ludo’.

RE: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Cook, Malcolm]

Hi!

> >> > Can anyone elaborate a little on what are the obstacles to having
> >> > `/gnu` mounted read-write network wide?
> >>
> >> Yes, the primary problem is that the daemon assumes that it is the
> >> only thing writing to the store and the localstatedir.  Any
> >> modification of profiles and the store goes through the daemon.
> >> > If so, might this be mitigated using a variant off of "Using the
> >> > Offload Facility"
> >> > (http://www.gnu.org/software/guix/manual/guix.html#Daemon-
> Offload-
> >> Setu
> >> > p) in which builds would still be offloaded (and thus subject to
> >> > coordination), with the elimination of the need for " Missing
> >> > prerequisites for the build are copied over SSH to the target
> >> > machine, which then proceeds with the build; upon success the
> >> > output(s) of the build are copied back to the initial machine"
> >> > since they would be done through the shared file system?
> >>
> >> Something like that has been suggested before: if the daemon were to
> >> accept authenticated connections from the outside rather than to just
> >> listen on a local socket we could have remote guix clients connecting
> >> to the central daemon.
> >
> > Hmm.  So, for now, I could just teach users to, i.e., `ssh
> ${USER}@${GUIX_HOST}   /path/to/guix package -r lua -i guile guile-cairo`
> and contrive for GUIX_HOST to be defined for them.
> >
> > Or, wrap it up even further, and contrive for everyone to have the
> following in their env:
> >
> > 	alias guixpkg='ssh ${USER}@${GUIX_HOST} /path/to/guix package'
> >
> > ??
> 
> That’s a possibility.

Thanks for the confirmation.  I'm thinking of trying this for the "short term" - unless Ricardo's efforts bear fruit sooner.

> Another one, which I think Ricardo has been investigating lately, would be to
> have users manage their profiles via the Web user interface,
> guix-web: <https://git.dthompson.us/guix-web.git>.  (The video
> <http://audio-video.gnu.org/video/misc/2015-
> 01__GNU_Guix__The_Emacs_of_Distros.webm>
> has a demo starting at around 23mn.)

Hmm, it is not clear to me how this would play out in a multi-user environment.  Having a web server that could alter per-user profiles sounds like a recipe for more confusion, rather than a solution.

In any case, in my opinion, it would be a mistake to have depend on another tool (such as guix-web, or something similar) to implement functionality that could not be gained at using command-line  guix.  Don't you agree?

> >> Currently our users are okay with that, probably to a large part
> >> because they are not yet aware of all the features of Guix.  They are
> >> only used to management by sysadmins or manual compilation, so they
> >> are not inconvenienced.
> >>
> >> Ultimately, the correct fix is to allow remote guix clients to
> >> communicate with a central guix daemon.  The daemon does not even
> >> need to be aware of remote connections if guix clients can
> >> transparently connect via SSH and send RPCs to the socket.  This is not yet
> implemented.
> >
> > Sounds great.  On the roadmap?
> 
> Definitely.  There are details to be sorted out (SSH vs. plain old socket), but
> we should discuss it and “get it done.”

Are there any signposts on this road-map emerging?  Is there any way I can help, such as review proposals for how the CLI would work, I'd be happy to chip in.  Not ready to start slinging guile though.  Maybe later (the old lisper in me revels at the thought).

> Thanks for your feedback,
> Ludo’.

Thanks for GUIX!

~Malcolm

Re: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Ricardo Wurmus]

>> Another one, which I think Ricardo has been investigating lately, would be to
>> have users manage their profiles via the Web user interface,
>> guix-web: <https://git.dthompson.us/guix-web.git>.  (The video
>> <http://audio-video.gnu.org/video/misc/2015-
>> 01__GNU_Guix__The_Emacs_of_Distros.webm>
>> has a demo starting at around 23mn.)
>
> Hmm, it is not clear to me how this would play out in a multi-user
> environment.  Having a web server that could alter per-user profiles
> sounds like a recipe for more confusion, rather than a solution.
>
> In any case, in my opinion, it would be a mistake to have depend on
> another tool (such as guix-web, or something similar) to implement
> functionality that could not be gained at using command-line guix.
> Don't you agree?

Some find a web interface easier to use than the command line.  For
those users I suspect the convenience of using a web interface outweighs
the threat of additional confusion.  Ultimately, I want both tools to be
available because users differ.

~~ Ricardo

Re: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Ludovic Courtès]

"Cook, Malcolm" <MEC@stowers.org> skribis:

>> Another one, which I think Ricardo has been investigating lately, would be to
>> have users manage their profiles via the Web user interface,
>> guix-web: <https://git.dthompson.us/guix-web.git>.  (The video
>> <http://audio-video.gnu.org/video/misc/2015-
>> 01__GNU_Guix__The_Emacs_of_Distros.webm>
>> has a demo starting at around 23mn.)
>
> Hmm, it is not clear to me how this would play out in a multi-user environment.  Having a web server that could alter per-user profiles sounds like a recipe for more confusion, rather than a solution.
>
> In any case, in my opinion, it would be a mistake to have depend on another tool (such as guix-web, or something similar) to implement functionality that could not be gained at using command-line  guix.  Don't you agree?

I’m just mentioning it as an additional option, for those who like it.

>> >> Ultimately, the correct fix is to allow remote guix clients to
>> >> communicate with a central guix daemon.  The daemon does not even
>> >> need to be aware of remote connections if guix clients can
>> >> transparently connect via SSH and send RPCs to the socket.  This is not yet
>> implemented.
>> >
>> > Sounds great.  On the roadmap?
>> 
>> Definitely.  There are details to be sorted out (SSH vs. plain old socket), but
>> we should discuss it and “get it done.”
>
> Are there any signposts on this road-map emerging?  Is there any way I can help, such as review proposals for how the CLI would work, I'd be happy to chip in.  Not ready to start slinging guile though.  Maybe later (the old lisper in me revels at the thought).

Yes, one way to help would be by trying a simple setup using Guix as it
currently exists, coupled with socat:

  http://debbugs.gnu.org/cgi/bugreport.cgi?bug=20381

If you could test this and provide feedback about the other options
discussed there, that would be great (please email
20381@debbugs.gnu.org.)

So no ETA, but once we have feedback from you, Ricardo, and other
interested parties, it will be easier to make a choice and implement it.
:-)

Thanks,
Ludo’.

RE: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Cook, Malcolm]

[-- Attachment #1: Type: text/plain, Size: 4891 bytes --]

Ludovic, and Ricardo,

> Yes, one way to help would be by trying a simple setup using Guix as it
> currently exists, coupled with socat:
> 
>   http://debbugs.gnu.org/cgi/bugreport.cgi?bug=20381
> 
> If you could test this and provide feedback about the other options discussed
> there, that would be great (please email
> 20381@debbugs.gnu.org.)

Excellent.  That definitely looks testable by me, once I'm done with my basic site deployment.  I've been moving rather slowly but might be ready to contrast such approach with simply calling guix on guix-master over ssh.  I'll plan on it.  Maybe a week or two out.

Reading the thread, it sounds like Ricardo has today already had good success with socat.  Yay!

In that thread, you suggested "What about installing Guix in /gnu/bin (say) and sharing it over NFS?".  Is there a way to `./bootstrap ./configure ` this?  Something like -- exec_prefix=/gnu --localstatedir=/gnu/var?  

Also, you discussed need for setting NIX_STATE_DIR.  This should not be needed if guix was already installed with configuration of --localstatedir=/guix/var   (with /guix being nfs mounted), right?

Ricardo also wrote "For ‘guix package’ to work, /gnu/var/guix/profiles/per-user must be shared read-write (over NFS) with correct UID mapping." - I really don't understand this and tried to elicit feedback in http://lists.nongnu.org/archive/html/guix-devel/2015-07/msg00216.html -   Can anyone give me clarification on this.  I though the guix daemon is going to manage these profiles.

Honestly, much of my understanding of the above comes from reading the manual, archived IRC chats and mail threads, FOSDEM vids, and blog posts - which has been great.  But,  I've yet to succeed in my own site deploy, which will give me testbed to stretch my own understanding.  Today, I had the following failure....

Ricardo, et. al.,  would you mind sharing some details on your network sharing configuration as regards users/groups and privileges, for I seem to have painted myself into a corner with the following
          
	/gnu  is an NFS mount, owned by network user 'guix', group-owned by  guix-builder with mod of 1775 
 
	guix is being built from git configured with -- exec_prefix=/gnu --localstatedir=/gnu/var 

	`make check`  results are

============================================================================
Testsuite summary for GNU Guix 0.8.3
============================================================================
# TOTAL: 42
# PASS:  32
# SKIP:  0
# XFAIL: 0
# FAIL:  10
# XPASS: 0
# ERROR: 0

With details below....   I can submit all this to bug-guix@gnu.org, but I suspect (1) there is something fundamentally wrong with the way I am setting things up (2) your are reading this anyway ;)

Can you see any gotchas I haven't got yet?  Can you see the error in my ways and suggest a way through?

FWIW, I try to `make install` anyway, and, I pause.... I cannot complete a make install without "changing hats".  I find that it works to do the following

		sudo make install-data # as root on localhost
		sudo -u guix make install-exec ## since root on localhost is squashed on the NFS mount point, 

But I don't know if this is complete or not (the Makefile is complex!).

Anyway, "changing hats" feels very wrong, and I can feel more trouble coming when I want to run the daemon....

Thanks so much everyone, and, keep on guixing,

~Malcolm


PS - in case it helps - below are test results and attached are tarred logs - I have not yet taught myself how to read them with understanding.... - (have a great weekend!)

PASS: tests/base32.scm
PASS: tests/base64.scm
FAIL: tests/cpio.scm
PASS: tests/hash.scm
PASS: tests/pk-crypto.scm
PASS: tests/pki.scm
PASS: tests/sets.scm
PASS: tests/substitute.scm
FAIL: tests/builders.scm
FAIL: tests/derivations.scm
PASS: tests/ui.scm
PASS: tests/records.scm
PASS: tests/utils.scm
PASS: tests/build-utils.scm
FAIL: tests/packages.scm
PASS: tests/snix.scm
PASS: tests/hackage.scm
PASS: tests/elpa.scm
FAIL: tests/store.scm
PASS: tests/monads.scm
PASS: tests/gexp.scm
FAIL: tests/nar.scm
FAIL: tests/union.scm
PASS: tests/profiles.scm
FAIL: tests/syscalls.scm
PASS: tests/gremlin.scm
PASS: tests/lint.scm
PASS: tests/publish.scm
PASS: tests/size.scm
PASS: tests/guix-build.sh
PASS: tests/guix-download.sh
PASS: tests/guix-hash.sh
PASS: tests/guix-package.sh
FAIL: tests/guix-package-net.sh
PASS: tests/guix-system.sh
PASS: tests/guix-archive.sh
PASS: tests/guix-authenticate.sh
FAIL: tests/guix-environment.sh
PASS: tests/guix-lint.sh
PASS: tests/guix-register.sh
PASS: tests/guix-gc.sh
PASS: tests/guix-daemon.sh



> So no ETA, but once we have feedback from you, Ricardo, and other
> interested parties, it will be easier to make a choice and implement it.
> :-)
> 
> Thanks,
> Ludo’.

[-- Attachment #2: guixtests.log --]
[-- Type: application/octet-stream, Size: 1525760 bytes --]

Test suite failures

[Ludovic Courtès]

(Stripping the list of recipients.)

"Cook, Malcolm" <MEC@stowers.org> skribis:

> In that thread, you suggested "What about installing Guix in /gnu/bin (say) and sharing it over NFS?".  Is there a way to `./bootstrap ./configure ` this?  Something like -- exec_prefix=/gnu --localstatedir=/gnu/var?  

Yes, --exec-prefix=/gnu --localstatedir=/gnu/var (note “--exec-prefix”
with a hyphen.)

> Also, you discussed need for setting NIX_STATE_DIR.  This should not be needed if guix was already installed with configuration of --localstatedir=/guix/var   (with /guix being nfs mounted), right?

Right.

> Ricardo also wrote "For ‘guix package’ to work, /gnu/var/guix/profiles/per-user must be shared read-write (over NFS) with correct UID mapping." - I really don't understand this and tried to elicit feedback in http://lists.nongnu.org/archive/html/guix-devel/2015-07/msg00216.html -   Can anyone give me clarification on this.  I though the guix daemon is going to manage these profiles.

‘guix package’ makes remote procedure calls (RPCs) to the daemon and the
daemon does most things on its behalf.

However, there’s one thing that ‘guix package’ does directly by itself:
updating the profile symlinks, ~/.guix-profile and its target, which is
in $localstatedir.

For that reason, ‘guix package’ must have write access to
$localstatedir.

Does that clarify things?

> ============================================================================
> Testsuite summary for GNU Guix 0.8.3
> ============================================================================
> # TOTAL: 42
> # PASS:  32
> # SKIP:  0
> # XFAIL: 0
> # FAIL:  10
> # XPASS: 0
> # ERROR: 0

[...]

> FAIL: tests/cpio.scm

What does ‘cpio --version’ returns?  Could you email me (privately) the
file ‘t.cpio’ made with these commands:

  cd /path/to/guix
  ln -s chbouib /tmp/somelink
  (for i in  / $PWD /tmp/somelink $PWD/guix.scm \
       $PWD/guix/build/syscalls.scm $PWD/guix/packages.scm ; \
       do echo $i ; done ) | \
       cpio -o -O t.cpio -H newc

> FAIL: tests/builders.scm
> FAIL: tests/derivations.scm

These two show errors like:

--8<---------------cut here---------------start------------->8---
output path `/home/mec/project/sce/guix/test-tmp/store/2l1h7am3q3c6sd4z5425mp3kjady6nla-hello-2.8.tar.gz should be a non-executable regular file
--8<---------------cut here---------------end--------------->8---

Which makes it sound like files are automatically made executable on
this file system.

What does ‘umask’ return?

What about ‘mount | grep /home’?

> FAIL: tests/packages.scm

Ditto.

> PASS: tests/snix.scm
> PASS: tests/hackage.scm
> PASS: tests/elpa.scm
> FAIL: tests/store.scm

This one shows this:

--8<---------------cut here---------------start------------->8---
;;; (c #<condition &nix-protocol-error [message: "path `/home/mec/project/sce/guix/test-tmp/stor?/vlryjv621i7xj4sr8ajzl9k9w5bmkv5c-text' is not in the Nix store" status: 1] 24a61e0>)
tests/store.scm:560: FAIL import corrupt path

;;; (verify1 #t)

;;; (verify2 #f)

;;; (verify3 #t)
tests/store.scm:620: FAIL verify-store + check-contents
--8<---------------cut here---------------end--------------->8---

This particular test is not designed to work with long $top_builddir
like this (the test corrupts an archive, but in this case the corruption
occurs in the middle of the file name, which confuses the test logic.)
I’ll see how to fix it.

> FAIL: tests/nar.scm

--8<---------------cut here---------------start------------->8---
tests/nar.scm:172: FAIL write-file puts file in C locale collation order
--8<---------------cut here---------------end--------------->8---

> FAIL: tests/union.scm

“Non-executable” error as before.

> PASS: tests/profiles.scm
> FAIL: tests/syscalls.scm

Harmless clone/setns/pivot-root test failures probably due to the fact
that the host kernel is too old.  Can you confirm that this is fixed in
‘master’ (commit b62a3eb)?  Simply run:

  make check TESTS=tests/syscalls.scm

> FAIL: tests/guix-package-net.sh

“Non-executable” error again.

> FAIL: tests/guix-environment.sh

Likewise.

Thanks for your help!

Ludo’.

RE: Test suite failures

[Cook, Malcolm]

Ludo,

As you observed, the majority of my problems were due to building guix on a network mount having some odd nfs4_facl setting that was making all the files be executable, which is explicitly tested against in many tests, and thus they were all failing.

Moving off that file system, and building in /tmp,  I am now building guix-0.8.2  on centOS7 passing all tests.

Thanks so much for reviewing these logs and your guidance.

I hope to be cooking with guix soon.

~Malcolm

> -----Original Message-----
> From: Ludovic Courtès [mailto:ludo@gnu.org]
> Sent: Monday, July 13, 2015 11:46 AM
> To: Cook, Malcolm
> Cc: Guix-devel
> Subject: Test suite failures
> 
> (Stripping the list of recipients.)
> 
> "Cook, Malcolm" <MEC@stowers.org> skribis:
> 
> > In that thread, you suggested "What about installing Guix in /gnu/bin (say)
> and sharing it over NFS?".  Is there a way to `./bootstrap ./configure ` this?
> Something like -- exec_prefix=/gnu --localstatedir=/gnu/var?
> 
> Yes, --exec-prefix=/gnu --localstatedir=/gnu/var (note “--exec-prefix”
> with a hyphen.)
> 
> > Also, you discussed need for setting NIX_STATE_DIR.  This should not be
> needed if guix was already installed with configuration of --
> localstatedir=/guix/var   (with /guix being nfs mounted), right?
> 
> Right.
> 
> > Ricardo also wrote "For ‘guix package’ to work, /gnu/var/guix/profiles/per-
> user must be shared read-write (over NFS) with correct UID mapping." - I
> really don't understand this and tried to elicit feedback in
> http://lists.nongnu.org/archive/html/guix-devel/2015-07/msg00216.html -
> Can anyone give me clarification on this.  I though the guix daemon is going to
> manage these profiles.
> 
> ‘guix package’ makes remote procedure calls (RPCs) to the daemon and the
> daemon does most things on its behalf.
> 
> However, there’s one thing that ‘guix package’ does directly by itself:
> updating the profile symlinks, ~/.guix-profile and its target, which is in
> $localstatedir.
> 
> For that reason, ‘guix package’ must have write access to $localstatedir.
> 
> Does that clarify things?
> 
> >
> ==========================================================
> ============
> > ======
> > Testsuite summary for GNU Guix 0.8.3
> >
> ==========================================================
> ============
> > ======
> > # TOTAL: 42
> > # PASS:  32
> > # SKIP:  0
> > # XFAIL: 0
> > # FAIL:  10
> > # XPASS: 0
> > # ERROR: 0
> 
> [...]
> 
> > FAIL: tests/cpio.scm
> 
> What does ‘cpio --version’ returns?  Could you email me (privately) the file
> ‘t.cpio’ made with these commands:
> 
>   cd /path/to/guix
>   ln -s chbouib /tmp/somelink
>   (for i in  / $PWD /tmp/somelink $PWD/guix.scm \
>        $PWD/guix/build/syscalls.scm $PWD/guix/packages.scm ; \
>        do echo $i ; done ) | \
>        cpio -o -O t.cpio -H newc
> 
> > FAIL: tests/builders.scm
> > FAIL: tests/derivations.scm
> 
> These two show errors like:
> 
> --8<---------------cut here---------------start------------->8---
> output path `/home/mec/project/sce/guix/test-
> tmp/store/2l1h7am3q3c6sd4z5425mp3kjady6nla-hello-2.8.tar.gz should be a
> non-executable regular file --8<---------------cut here---------------end----------
> ----->8---
> 
> Which makes it sound like files are automatically made executable on this file
> system.
> 
> What does ‘umask’ return?
> 
> What about ‘mount | grep /home’?
> 
> > FAIL: tests/packages.scm
> 
> Ditto.
> 
> > PASS: tests/snix.scm
> > PASS: tests/hackage.scm
> > PASS: tests/elpa.scm
> > FAIL: tests/store.scm
> 
> This one shows this:
> 
> --8<---------------cut here---------------start------------->8---
> ;;; (c #<condition &nix-protocol-error [message: "path
> `/home/mec/project/sce/guix/test-
> tmp/stor?/vlryjv621i7xj4sr8ajzl9k9w5bmkv5c-text' is not in the Nix store"
> status: 1] 24a61e0>)
> tests/store.scm:560: FAIL import corrupt path
> 
> ;;; (verify1 #t)
> 
> ;;; (verify2 #f)
> 
> ;;; (verify3 #t)
> tests/store.scm:620: FAIL verify-store + check-contents --8<---------------cut
> here---------------end--------------->8---
> 
> This particular test is not designed to work with long $top_builddir like this
> (the test corrupts an archive, but in this case the corruption occurs in the
> middle of the file name, which confuses the test logic.) I’ll see how to fix it.
> 
> > FAIL: tests/nar.scm
> 
> --8<---------------cut here---------------start------------->8---
> tests/nar.scm:172: FAIL write-file puts file in C locale collation order --8<-------
> --------cut here---------------end--------------->8---
> 
> > FAIL: tests/union.scm
> 
> “Non-executable” error as before.
> 
> > PASS: tests/profiles.scm
> > FAIL: tests/syscalls.scm
> 
> Harmless clone/setns/pivot-root test failures probably due to the fact that
> the host kernel is too old.  Can you confirm that this is fixed in ‘master’
> (commit b62a3eb)?  Simply run:
> 
>   make check TESTS=tests/syscalls.scm
> 
> > FAIL: tests/guix-package-net.sh
> 
> “Non-executable” error again.
> 
> > FAIL: tests/guix-environment.sh
> 
> Likewise.
> 
> Thanks for your help!
> 
> Ludo’.

Re: Test suite failures

[Ludovic Courtès]

"Cook, Malcolm" <MEC@stowers.org> skribis:

> As you observed, the majority of my problems were due to building guix on a network mount having some odd nfs4_facl setting that was making all the files be executable, which is explicitly tested against in many tests, and thus they were all failing.
>
> Moving off that file system, and building in /tmp,  I am now building guix-0.8.2  on centOS7 passing all tests.

That’s good news.  It would be great though if you could run the tests
on ‘master’ as well, like you did before, to make sure all the issues
really vanished.

Thanks,
Ludo’.

RE: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Cook, Malcolm]

Ludovic, and Ricardo,

> Yes, one way to help would be by trying a simple setup using Guix as it
> currently exists, coupled with socat:
> 
>   http://debbugs.gnu.org/cgi/bugreport.cgi?bug=20381
> 
> If you could test this and provide feedback about the other options discussed
> there, that would be great (please email
> 20381@debbugs.gnu.org.)

Excellent.  That definitely looks testable by me, once I'm done with my basic site deployment.  I've been moving rather slowly but might be ready to contrast such approach with simply calling guix on guix-master over ssh.  I'll plan on it.  Maybe a week or two out.

Reading the thread, it sounds like Ricardo has today already had good success with socat.  Yay!

In that thread, you suggested "What about installing Guix in /gnu/bin (say) and sharing it over NFS?".  Is there a way to `./bootstrap ./configure ` this?  Something like -- exec_prefix=/gnu --localstatedir=/gnu/var?  

Also, you discussed need for setting NIX_STATE_DIR.  This should not be needed if guix was already installed with configuration of --localstatedir=/guix/var   (with /guix being nfs mounted), right?

Ricardo also wrote "For ‘guix package’ to work, /gnu/var/guix/profiles/per-user must be shared read-write (over NFS) with correct UID mapping." - I really don't understand this and tried to elicit feedback in http://lists.nongnu.org/archive/html/guix-devel/2015-07/msg00216.html -   Can anyone give me clarification on this.  I though the guix daemon is going to manage these profiles.

Honestly, much of my understanding of the above comes from reading the manual, archived IRC chats and mail threads, FOSDEM vids, and blog posts - which has been great.  But,  I've yet to succeed in my own site deploy, which will give me testbed to stretch my own understanding.  Today, I had the following failure....

Ricardo, et. al.,  would you mind sharing some details on your network sharing configuration as regards users/groups and privileges, for I seem to have painted myself into a corner with the following
          
	/gnu  is an NFS mount, owned by network user 'guix', group-owned by  guix-builder with mod of 1775 
 
	guix is being built from git configured with -- exec_prefix=/gnu --localstatedir=/gnu/var 

	`make check`  results are

============================================================================
Testsuite summary for GNU Guix 0.8.3
============================================================================
# TOTAL: 42
# PASS:  32
# SKIP:  0
# XFAIL: 0
# FAIL:  10
# XPASS: 0
# ERROR: 0

With details below....   I can submit all this to bug-guix@gnu.org, but I suspect (1) there is something fundamentally wrong with the way I am setting things up (2) your are reading this anyway ;)

Can you see any gotchas I haven't got yet?  Can you see the error in my ways and suggest a way through?

FWIW, I try to `make install` anyway, and, I pause.... I cannot complete a make install without "changing hats".  I find that it works to do the following

		sudo make install-data # as root on localhost
		sudo -u guix make install-exec ## since root on localhost is squashed on the NFS mount point, 

But I don't know if this is complete or not (the Makefile is complex!).

Anyway, "changing hats" feels very wrong, and I can feel more trouble coming when I want to run the daemon....

Thanks so much everyone, and, keep on guixing,

~Malcolm


PS - in case it helps - below are test results and tarred logs are at http://research.stowers.org/mec/guixtests.tar

 I have not yet taught myself how to read them with understanding.... - (have a great weekend!)
	
PASS: tests/base32.scm
PASS: tests/base64.scm
FAIL: tests/cpio.scm
PASS: tests/hash.scm
PASS: tests/pk-crypto.scm
PASS: tests/pki.scm
PASS: tests/sets.scm
PASS: tests/substitute.scm
FAIL: tests/builders.scm
FAIL: tests/derivations.scm
PASS: tests/ui.scm
PASS: tests/records.scm
PASS: tests/utils.scm
PASS: tests/build-utils.scm
FAIL: tests/packages.scm
PASS: tests/snix.scm
PASS: tests/hackage.scm
PASS: tests/elpa.scm
FAIL: tests/store.scm
PASS: tests/monads.scm
PASS: tests/gexp.scm
FAIL: tests/nar.scm
FAIL: tests/union.scm
PASS: tests/profiles.scm
FAIL: tests/syscalls.scm
PASS: tests/gremlin.scm
PASS: tests/lint.scm
PASS: tests/publish.scm
PASS: tests/size.scm
PASS: tests/guix-build.sh
PASS: tests/guix-download.sh
PASS: tests/guix-hash.sh
PASS: tests/guix-package.sh
FAIL: tests/guix-package-net.sh
PASS: tests/guix-system.sh
PASS: tests/guix-archive.sh
PASS: tests/guix-authenticate.sh
FAIL: tests/guix-environment.sh
PASS: tests/guix-lint.sh
PASS: tests/guix-register.sh
PASS: tests/guix-gc.sh
PASS: tests/guix-daemon.sh



> So no ETA, but once we have feedback from you, Ricardo, and other
> interested parties, it will be easier to make a choice and implement it.
> :-)
> 
> Thanks,
> Ludo’.

Re: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Ricardo Wurmus]

Cook, Malcolm <MEC@stowers.org> writes:

> Also, you discussed need for setting NIX_STATE_DIR.  This should not
> be needed if guix was already installed with configuration of
> --localstatedir=/guix/var (with /guix being nfs mounted), right?

NIX_STATE_DIR is used to override localstate at runtime.  If you are
happy with the localstatedir (defined at configure time) you do not need
NIX_STATE_DIR.  After my vacation I’ll take some time to think about how
to allow users to run “guix” on cluster nodes to manipulate their
profiles — and how to reliably set this all up.  I’m planning on
documenting this, both in a specialised form for my blog and in a more
generic fashion for the Guix manual.

> Ricardo also wrote "For ‘guix package’ to work,
> /gnu/var/guix/profiles/per-user must be shared read-write (over NFS)
> with correct UID mapping." - I really don't understand this and tried
> to elicit feedback in
> http://lists.nongnu.org/archive/html/guix-devel/2015-07/msg00216.html
> - Can anyone give me clarification on this.  I though the guix daemon
> is going to manage these profiles.

I don’t remember writing this or the context in which I wrote this
(could it be that Ludovic wrote this?) — looking at the code for
guix/scripts/package.scm it seems that the client performs some of the
symlink switching when switching generations (e.g. after installing a
new package).  For that to work it would need write permission to the
user’s profile directory.

(I could be completely wrong here.  That’s not how it’s set up here and
I haven’t yet tested a configuration like this.)

~~ Ricardo

RE: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Cook, Malcolm]

> Cook, Malcolm <MEC@stowers.org> writes:
> 
> > Also, you discussed need for setting NIX_STATE_DIR.  This should not
> > be needed if guix was already installed with configuration of
> > --localstatedir=/guix/var (with /guix being nfs mounted), right?
> 
> NIX_STATE_DIR is used to override localstate at runtime.  If you are happy
> with the localstatedir (defined at configure time) you do not need
> NIX_STATE_DIR.  After my vacation I’ll take some time to think about how to
> allow users to run “guix” on cluster nodes to manipulate their profiles — and
> how to reliably set this all up.  I’m planning on documenting this, both in a
> specialised form for my blog and in a more generic fashion for the Guix
> manual.

Great - thanks for clarifying - I am looking forward to reading about this further in both places.  

I will eventually care about the cluster use case, but for now my first aim is to ensure that users at my institute can expect an identical experience in using guix regardless of which of our multiple shared-used computational servers they are logged into.

> 
> > Ricardo also wrote "For ‘guix package’ to work,
> > /gnu/var/guix/profiles/per-user must be shared read-write (over NFS)
> > with correct UID mapping." - I really don't understand this and tried
> > to elicit feedback in
> > http://lists.nongnu.org/archive/html/guix-devel/2015-07/msg00216.html
> > - Can anyone give me clarification on this.  I though the guix daemon
> > is going to manage these profiles.
> 
> I don’t remember writing this or the context in which I wrote this (could it be
> that Ludovic wrote this?) — looking at the code for guix/scripts/package.scm
> it seems that the client performs some of the symlink switching when
> switching generations (e.g. after installing a new package).  For that to work it
> would need write permission to the user’s profile directory.

Ricardo I am sorry, you are correct, I was quoting Ludovic in a thread where you replied https://lists.gnu.org/archive/html/bug-guix/2015-07/msg00042.html 

> (I could be completely wrong here.  That’s not how it’s set up here and I
> haven’t yet tested a configuration like this.)

I hope to chime in further with a big WOOHOO once I have this configuration working....

Thanks again for your helps...

~Malcolm

> 
> ~~ Ricardo

Re: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Pjotr Prins]

Great Malcolm,

One thing I was advocating today (compared to, for example, easybuild,
modules or homebrew) is that once you create a GNU Guix package it
will live on *all* future Linux systems (and perhaps beyond). It being
a GNU project I can only imagine it becoming ubiquous.

Pj.

On Wed, Jul 15, 2015 at 07:49:53PM +0000, Cook, Malcolm wrote:
> > Cook, Malcolm <MEC@stowers.org> writes:
> > 
> > > Also, you discussed need for setting NIX_STATE_DIR.  This should not
> > > be needed if guix was already installed with configuration of
> > > --localstatedir=/guix/var (with /guix being nfs mounted), right?
> > 
> > NIX_STATE_DIR is used to override localstate at runtime.  If you are happy
> > with the localstatedir (defined at configure time) you do not need
> > NIX_STATE_DIR.  After my vacation I’ll take some time to think about how to
> > allow users to run “guix” on cluster nodes to manipulate their profiles — and
> > how to reliably set this all up.  I’m planning on documenting this, both in a
> > specialised form for my blog and in a more generic fashion for the Guix
> > manual.
> 
> Great - thanks for clarifying - I am looking forward to reading about this further in both places.  
> 
> I will eventually care about the cluster use case, but for now my first aim is to ensure that users at my institute can expect an identical experience in using guix regardless of which of our multiple shared-used computational servers they are logged into.
> 
> > 
> > > Ricardo also wrote "For ‘guix package’ to work,
> > > /gnu/var/guix/profiles/per-user must be shared read-write (over NFS)
> > > with correct UID mapping." - I really don't understand this and tried
> > > to elicit feedback in
> > > http://lists.nongnu.org/archive/html/guix-devel/2015-07/msg00216.html
> > > - Can anyone give me clarification on this.  I though the guix daemon
> > > is going to manage these profiles.
> > 
> > I don’t remember writing this or the context in which I wrote this (could it be
> > that Ludovic wrote this?) — looking at the code for guix/scripts/package.scm
> > it seems that the client performs some of the symlink switching when
> > switching generations (e.g. after installing a new package).  For that to work it
> > would need write permission to the user’s profile directory.
> 
> Ricardo I am sorry, you are correct, I was quoting Ludovic in a thread where you replied https://lists.gnu.org/archive/html/bug-guix/2015-07/msg00042.html 
> 
> > (I could be completely wrong here.  That’s not how it’s set up here and I
> > haven’t yet tested a configuration like this.)
> 
> I hope to chime in further with a big WOOHOO once I have this configuration working....
> 
> Thanks again for your helps...
> 
> ~Malcolm
> 
> > 
> > ~~ Ricardo
> 

-- 

RE: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Cook, Malcolm]

> I hope to chime in further with a big WOOHOO once I have this configuration
> working....

Not quite WOOHOO yet.  Getting close....  Here is where I am:

I have built guix-0.8.2 on centOS7 passing all checks

However

	./pre-inst-env guix package -i hello

Gives me

	guix package: error: build failed: the build users group `guix-builder' has no members

But, but, I do have a group `guix-builder`

I think the problem may be that my guix-builder group was defined in NIS whereas its members are not in NIS.

I don't know the method that guix is using to detect group membership, but my configuration seems to not respect it.

I am going to try putting guix-builder1-10 in NIS also and see if that helps, but, while I wait for my sysadmin to help me, I thought I'd ask all y'all.


My setup now is like this:


/gnu is network shared mount rw everywhere 
	owned by NIS user guix 
	group owned by NIS group guix-builder

I configured the build with --prefix=/gnu

All tests pass.

The daemon is running as user `guix` on host  ${GUIXHOST}  (I welcome a better suggestion for this host.  Maybe just GUIX is better?  Or GUIXMASTER?)

Build works:

	> guix build hello
	...
	The following files will be downloaded:
	   /gnu/store/ydj867rdlz3p6lnnmbqn7jhvdf3i0dvi-hello-2.10
	   /gnu/store/hy2hi0zj5hrqkmkhpdxf04c9bcnlnsf9-glibc-2.21
	...
 	603 operations

And confirmed:

	/gnu/store/ydj867rdlz3p6lnnmbqn7jhvdf3i0dvi-hello-2.10/bin/hello
	Hello, world

However trying to --install is not successful

	/gnu/bin/guix package --install hello
	accepted connection from pid 26597, uid 1232
	error: while creating directory `/gnu/var/guix/profiles/per-user/mec': Permission denied
	Please create the `/gnu/var/guix/profiles/per-user/mec' directory, with you as the owner.
	Try "info '(guix) Invoking guix package'" for more information.

Which leads me to think the documentation is in error where it says "The per-user directory is created when guix-daemon is started".  

Creating it by hand (owned by guix, with write access to all) :

	mkdir /gnu/var/guix/profiles/per-user ## the doc says this should happen by the daemon but not!  FIXME! BUG?
	chmod a+rwx /gnu/var/guix/profiles/per-user

gets me further, and  I did find the rest of the sentence to be correct: " and the user sub-directory is created by guix package."

But --installing only gets a little further:
	/gnu/bin/guix package --install hello
	accepted connection from pid 31645, uid 1232
	The following package will be installed:
   	hello	2.10	/gnu/store/ydj867rdlz3p6lnnmbqn7jhvdf3i0dvi-hello-2.10

	The following derivations will be built:
	   /gnu/store/6cs8g6kq96zdwl25929l5s5y6gxk7hxs-profile.drv
	   /gnu/store/b1f6jwficry84qgigvkldkfyhlc2xhg2-ca-certificate-bundle.drv
	   /gnu/store/n2ks4dzmrcg9ghhrp6c6wsp57nib5rqf-info-dir.drv
	The following files will be downloaded:
 	  /gnu/store/kbf46vds6f7lp723xn62kgdaxmwc1jvi-module-import-compiled
 	  /gnu/store/41wzbfhmb6j5yha508y30kwycivhpx9b-module-import
	   /gnu/store/ki6mvjqbf3nc8lfavwqfsrxqzi7qhf50-module-import
	   /gnu/store/jhbkpizpakgk5xzzpyrh15vaprqcarb8-module-import-compiled
	guix package: error: build failed: the build users group `guix-builder' has no members
	

Huh? Why 
	a) is it going to download and build derivations AGAIN !?!?  
	b) does it tell me "the build users group `guix-builder' has no members"

So, I am stymied, and welcome your kind assistance and observations.

I feel a YAHOO WAHOO coming on,

Malcolm

Re: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Ludovic Courtès]

"Cook, Malcolm" <MEC@stowers.org> skribis:

> However
>
> 	./pre-inst-env guix package -i hello
>
> Gives me
>
> 	guix package: error: build failed: the build users group `guix-builder' has no members
>
> But, but, I do have a group `guix-builder`
>
> I think the problem may be that my guix-builder group was defined in NIS whereas its members are not in NIS.

What does ‘getenv group guix-builder’ return?  It should show something
like:

  guixbuild:x:30000:guixbuilder01,guixbuilder02,guixbuilder03,guixbuilder04,guixbuilder05,guixbuilder06,guixbuilder07,guixbuilder08,guixbuilder09,guixbuilder10

Each build account must have ‘guix-builder’ in its supplementary group list.

> The daemon is running as user `guix` on host  ${GUIXHOST}

The daemon really needs to run as root, if possible:

  http://www.gnu.org/software/guix/manual/html_node/Build-Environment-Setup.html

> However trying to --install is not successful
>
> 	/gnu/bin/guix package --install hello
> 	accepted connection from pid 26597, uid 1232
> 	error: while creating directory `/gnu/var/guix/profiles/per-user/mec': Permission denied
> 	Please create the `/gnu/var/guix/profiles/per-user/mec' directory, with you as the owner.
> 	Try "info '(guix) Invoking guix package'" for more information.
>
> Which leads me to think the documentation is in error where it says "The per-user directory is created when guix-daemon is started".  

Well, /var/guix/profiles/per-user is created by the daemon, but .../mec
is created by ‘guix package’.

> Creating it by hand (owned by guix, with write access to all) :
>
> 	mkdir /gnu/var/guix/profiles/per-user ## the doc says this should happen by the daemon but not!  FIXME! BUG?
> 	chmod a+rwx /gnu/var/guix/profiles/per-user

Perfect.

The daemon really does create /gnu/var/guix/profiles/per-user, making it
world-writable (see nix/libstore/local-store.cc:254.)

However, you mentioned guix-daemon was *not* running as root, in which
case it does not attempt to create this directory.  That could be the
reason.

Thanks for your report,
Ludo’.

Re: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Claes Wallin (韋嘉誠)]

[-- Attachment #1: Type: text/plain, Size: 1061 bytes --]

On Jul 18, 2015 5:13 PM, "Ludovic Courtès" <ludo@gnu.org> wrote:
> "Cook, Malcolm" <MEC@stowers.org> skribis:
>
> > However
> >
> >       ./pre-inst-env guix package -i hello
> >
> > Gives me
> >
> >       guix package: error: build failed: the build users group
`guix-builder' has no members
> >
> > But, but, I do have a group `guix-builder`
> >
> > I think the problem may be that my guix-builder group was defined in
NIS whereas its members are not in NIS.
>
> What does ‘getenv group guix-builder’ return?  It should show something
> like:

Typo: getent, not getenv.

>
guixbuild:x:30000:guixbuilder01,guixbuilder02,guixbuilder03,guixbuilder04,guixbuilder05,guixbuilder06,guixbuilder07,guixbuilder08,guixbuilder09,guixbuilder10
>
> Each build account must have ‘guix-builder’ in its supplementary group
list.
>
> > The daemon is running as user `guix` on host  ${GUIXHOST}
>
> The daemon really needs to run as root, if possible:
>
>
http://www.gnu.org/software/guix/manual/html_node/Build-Environment-Setup.html

[-- Attachment #2: Type: text/html, Size: 1524 bytes --]

Re: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Andreas Enge]

On Sun, Jul 19, 2015 at 11:18:34AM +0200, Claes Wallin (韋嘉誠) wrote:
> > > But, but, I do have a group `guix-builder`
> guixbuild:x:30000:guixbuilder01,guixbuilder02,guixbuilder03,guixbuilder04,guixbuilder05,guixbuilder06,guixbuilder07,guixbuilder08,guixbuilder09,guixbuilder10

The documentation on the names of the group and the build users changed
recently; maybe your set-up corresponds to the former documentation.
Then you should start the daemon with
   guix-daemon --build-users-group=guixbuild

Andreas

RE: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Cook, Malcolm]

> > However
> >
> > 	./pre-inst-env guix package -i hello
> >
> > Gives me
> >
> > 	guix package: error: build failed: the build users group
> > `guix-builder' has no members
> >
> > But, but, I do have a group `guix-builder`
> >
> > I think the problem may be that my guix-builder group was defined in NIS
> whereas its members are not in NIS.
> 
> What does ‘getenv group guix-builder’ return?  It should show something
> like: 
> guixbuild:x:30000:guixbuilder01,guixbuilder02,guixbuilder03,guixbuilder04,gu
> ixbuilder05,guixbuilder06,guixbuilder07,guixbuilder08,guixbuilder09,guixbuild
> er10

Answered below....

> Each build account must have ‘guix-builder’ in its supplementary group list.
> 
> > The daemon is running as user `guix` on host  ${GUIXHOST}
> 
> The daemon really needs to run as root, if possible:
> 
>   http://www.gnu.org/software/guix/manual/html_node/Build-
> Environment-Setup.html

I guess my issues are arising from challenging this (soft?) requirement.

My SA has provided me /gnu as NFS network share owned by network user `guix` and group owned by `guix-builder`.

However it is 'root squashed' on all servers, including my GUIX_HOST.  In other works, the root account cannot write to it.

For this reason, I have tried 

	./configure --prefix=/gnu

rather than the suggested

	./configure --localstatedir=/gnu/var --exec-prefix=/gnu  

Since this suggestion results in `make install` needed to write to both the root-squashed network share and /usr/local which can only be done by root, an impossibility.

Or am I thinking about this wrong?

--prefix=/gnu has some additional advantage of getting the /share directory network wide, as presumably should be /etc/bashy_completion.d 

I am making guix from fresh `git pull` master passing all tests (except for SKIPPING the container test thanks to recent commit - last week it was failing this one)

However, with this configuration, after install, and running the daemon as user 'guix', I now get:

	bash-4.2$ guix build hello
	guix build: error: build failed: acquiring/releasing lock: No locks available

Oh, right, and, I am still doing by hand:
	su -c 'mkdir /gnu/var/guix/profiles/per-user; chmod a+rwx /gnu/var/guix/profiles/per-user' - guix

So, I think if I knew a little more about guix internals, I would expect that I could figure out a ./configuration that allows /gnu to reside on root-squashed network share.

Ricardo, you seem to have something _like_ this working.  I'm guessing that your /gnu was local to your GUIXHOST, and so you did not need to solve this root squash issue.  

Ricard & Ludo, do you mind thinking this through a little further with me?

Here is current my account setup

	getent group guix-builder
	guix-builder:!:20302:guix-builder1,guix-builder2,guix-builder3,guix-builder4,guix-builder5,guix-builder6,guix-builder7,guix-builder8,guix-builder9,guix-builder10

	getent passwd guix
	guix:$1$E5Ru3NpE$wZZY.cM8TwbRMHBI1UP110:3036:20302:Guix build user:/var/empty:/bin/bash

	getent passwd guix-builder1 ## and all the other guix-builder
	guix-builder1:!!:3048:20302:Guix build user 1:/var/empty:/sbin/nologin


~Malcolm


> 
> > However trying to --install is not successful
> >
> > 	/gnu/bin/guix package --install hello
> > 	accepted connection from pid 26597, uid 1232
> > 	error: while creating directory `/gnu/var/guix/profiles/per-
> user/mec': Permission denied
> > 	Please create the `/gnu/var/guix/profiles/per-user/mec' directory,
> with you as the owner.
> > 	Try "info '(guix) Invoking guix package'" for more information.
> >
> > Which leads me to think the documentation is in error where it says "The
> per-user directory is created when guix-daemon is started".
> 
> Well, /var/guix/profiles/per-user is created by the daemon, but .../mec is
> created by ‘guix package’.
> 
> > Creating it by hand (owned by guix, with write access to all) :
> >
> > 	mkdir /gnu/var/guix/profiles/per-user ## the doc says this should
> happen by the daemon but not!  FIXME! BUG?
> > 	chmod a+rwx /gnu/var/guix/profiles/per-user
> 
> Perfect.
> 
> The daemon really does create /gnu/var/guix/profiles/per-user, making it
> world-writable (see nix/libstore/local-store.cc:254.)
> 
> However, you mentioned guix-daemon was *not* running as root, in which
> case it does not attempt to create this directory.  That could be the reason. 
> 
> Thanks for your report,
> Ludo’.

RE: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Cook, Malcolm]

Hi,

I have resolved a main issue by having /gnu mounted without root squashing and running gnu-daemon as root, as strongly suggested after building with `./configure --localstatedir=/gnu/var --exec-prefix=/gnu`

And .... I am proceeding largely happily.

So.... WOOHOO and WHOOWHEE and general applause, cheers, and well-wishing from all quarters.

Thanks for everyone's advice and help.

I will start new threads with other issues.

Ludo, did I tell you, as you asked, I did build guix from master also without any FAILS and only SKIPPED one test, containers.scm.

~Malcolm



> -----Original Message-----
> From: Cook, Malcolm
> Sent: Monday, July 20, 2015 5:38 PM
> To: 'Ludovic Courtès'
> Cc: 'Ricardo Wurmus'; 'Guix-devel'; 'Pjotr Prins'; 'bio-
> packaging@mailman.open-bio.org'
> Subject: RE: Using a shared Guix store (was RE: [Bio-packaging] testing out
> guix)
> 
> > > However
> > >
> > > 	./pre-inst-env guix package -i hello
> > >
> > > Gives me
> > >
> > > 	guix package: error: build failed: the build users group
> > > `guix-builder' has no members
> > >
> > > But, but, I do have a group `guix-builder`
> > >
> > > I think the problem may be that my guix-builder group was defined in
> > > NIS
> > whereas its members are not in NIS.
> >
> > What does ‘getenv group guix-builder’ return?  It should show
> > something
> > like:
> > guixbuild:x:30000:guixbuilder01,guixbuilder02,guixbuilder03,guixbuilde
> > r04,gu
> > ixbuilder05,guixbuilder06,guixbuilder07,guixbuilder08,guixbuilder09,gu
> > ixbuild
> > er10
> 
> Answered below....
> 
> > Each build account must have ‘guix-builder’ in its supplementary group list.
> >
> > > The daemon is running as user `guix` on host  ${GUIXHOST}
> >
> > The daemon really needs to run as root, if possible:
> >
> >   http://www.gnu.org/software/guix/manual/html_node/Build-
> > Environment-Setup.html
> 
> I guess my issues are arising from challenging this (soft?) requirement.
> 
> My SA has provided me /gnu as NFS network share owned by network user
> `guix` and group owned by `guix-builder`.
> 
> However it is 'root squashed' on all servers, including my GUIX_HOST.  In
> other works, the root account cannot write to it.
> 
> For this reason, I have tried
> 
> 	./configure --prefix=/gnu
> 
> rather than the suggested
> 
> 	./configure --localstatedir=/gnu/var --exec-prefix=/gnu
> 
> Since this suggestion results in `make install` needed to write to both the
> root-squashed network share and /usr/local which can only be done by root,
> an impossibility.
> 
> Or am I thinking about this wrong?
> 
> --prefix=/gnu has some additional advantage of getting the /share directory
> network wide, as presumably should be /etc/bashy_completion.d
> 
> I am making guix from fresh `git pull` master passing all tests (except for
> SKIPPING the container test thanks to recent commit - last week it was failing
> this one)
> 
> However, with this configuration, after install, and running the daemon as
> user 'guix', I now get:
> 
> 	bash-4.2$ guix build hello
> 	guix build: error: build failed: acquiring/releasing lock: No locks
> available
> 
> Oh, right, and, I am still doing by hand:
> 	su -c 'mkdir /gnu/var/guix/profiles/per-user; chmod a+rwx
> /gnu/var/guix/profiles/per-user' - guix
> 
> So, I think if I knew a little more about guix internals, I would expect that I
> could figure out a ./configuration that allows /gnu to reside on root-squashed
> network share.
> 
> Ricardo, you seem to have something _like_ this working.  I'm guessing that
> your /gnu was local to your GUIXHOST, and so you did not need to solve this
> root squash issue.
> 
> Ricard & Ludo, do you mind thinking this through a little further with me?
> 
> Here is current my account setup
> 
> 	getent group guix-builder
> 	guix-builder:!:20302:guix-builder1,guix-builder2,guix-builder3,guix-
> builder4,guix-builder5,guix-builder6,guix-builder7,guix-builder8,guix-
> builder9,guix-builder10
> 
> 	getent passwd guix
> 	guix:$1$E5Ru3NpE$wZZY.cM8TwbRMHBI1UP110:3036:20302:Guix
> build user:/var/empty:/bin/bash
> 
> 	getent passwd guix-builder1 ## and all the other guix-builder
> 	guix-builder1:!!:3048:20302:Guix build user
> 1:/var/empty:/sbin/nologin
> 
> 
> ~Malcolm
> 
> 
> >
> > > However trying to --install is not successful
> > >
> > > 	/gnu/bin/guix package --install hello
> > > 	accepted connection from pid 26597, uid 1232
> > > 	error: while creating directory `/gnu/var/guix/profiles/per-
> > user/mec': Permission denied
> > > 	Please create the `/gnu/var/guix/profiles/per-user/mec' directory,
> > with you as the owner.
> > > 	Try "info '(guix) Invoking guix package'" for more information.
> > >
> > > Which leads me to think the documentation is in error where it says
> > > "The
> > per-user directory is created when guix-daemon is started".
> >
> > Well, /var/guix/profiles/per-user is created by the daemon, but
> > .../mec is created by ‘guix package’.
> >
> > > Creating it by hand (owned by guix, with write access to all) :
> > >
> > > 	mkdir /gnu/var/guix/profiles/per-user ## the doc says this should
> > happen by the daemon but not!  FIXME! BUG?
> > > 	chmod a+rwx /gnu/var/guix/profiles/per-user
> >
> > Perfect.
> >
> > The daemon really does create /gnu/var/guix/profiles/per-user, making
> > it world-writable (see nix/libstore/local-store.cc:254.)
> >
> > However, you mentioned guix-daemon was *not* running as root, in
> which
> > case it does not attempt to create this directory.  That could be the reason.
> >
> > Thanks for your report,
> > Ludo’.

Re: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Ricardo Wurmus]

> I have resolved a main issue by having /gnu mounted without root
> squashing and running gnu-daemon as root, as strongly suggested after
> building with `./configure --localstatedir=/gnu/var
> --exec-prefix=/gnu`
>
> And .... I am proceeding largely happily.
>
> So.... WOOHOO and WHOOWHEE and general applause, cheers, and well-wishing from all quarters.

Woo!  That’s great!
Glad it’s working for you.

~~ Ricardo

Re: Using a shared Guix store (was RE: [Bio-packaging] testing out guix)

[Ludovic Courtès]

"Cook, Malcolm" <MEC@stowers.org> skribis:

> So.... WOOHOO and WHOOWHEE and general applause, cheers, and well-wishing from all quarters.

Yay!  :-)

> Ludo, did I tell you, as you asked, I did build guix from master also without any FAILS and only SKIPPED one test, containers.scm.

Excellent.

Thanks for the joyful feedback!

Ludo’.