Re: guix on nfs based systems

Re: guix on nfs based systems

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2286 bytes --]

On Wed, Nov 22, 2023 at 05:32:07PM +0000, Etienne B. Roesch wrote:
> Hi,
> 
> I have been successful in convincing our IT dept to provide guix as
> standard on our vms and research clusters. We use this sophisticated
> platform that allows users to spawn a vm at will, and also use this shared
> ldap user database to allow for people to log on from computers/terminals
> spread on campus. Home directories are basically pulled from the OS (mostly
> ubuntu and centos) as a shared file system on the network, which then
> synchronises changes live. Practically, that means a user will find its
> desktop, preferences and files as they move to different computers.
> 
> Theoretically, a user can connect to several computers at the same time,
> which can lead to conflict issues at times, e.g. firefox doesn't like
> parallel access to its preferences. I haven't fully tested it, but we think
> that a guix profile is specific to a given computer, because of the way it
> links to the store, which is (currently) local to the vm, which isn't
> ideal: the whole point of using vms is for users to kill them when they
> don't need them.
> 
> What would be the recommended way of solving this? Has anybody had a
> similar situation?
> I am thinking we could either:
> - leave it as is, and train users to recreate their profiles whenever they
> use a new vm/computer (after all, that's what guix is for) but it could
> take some time to recompile if binaries aren't available
> - maybe turn /gnu/store as a shared nfs folder

CCing guix-science

One thing we do on the small cluster at UTenn is /gnu and /var/guix (in
addition to the home directories) are exported across NFS and then any
profiles that are installed in the users' home directories have their
symlinks not dangling.

I also have a snippet in my .profile to use GUIX_DAEMON_SOCKET to ssh
back to the head node to use the guix-daemon there, which keeps all the
guix stuff on one machine and easier to manage and make sure the same
software is available on all the machines.

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: guix on nfs based systems

[Etienne B. Roesch]

[-- Attachment #1: Type: text/plain, Size: 2660 bytes --]

Thanks! That's really helpful! How robust is your system to bandwidth
fluctuations and network hiccups?
How big is your store, /gnu, /var/guix?

How many users do you have?

Etienne

On Thu, Nov 23, 2023 at 6:32 AM Efraim Flashner <efraim@flashner.co.il>
wrote:

> On Wed, Nov 22, 2023 at 05:32:07PM +0000, Etienne B. Roesch wrote:
> > Hi,
> >
> > I have been successful in convincing our IT dept to provide guix as
> > standard on our vms and research clusters. We use this sophisticated
> > platform that allows users to spawn a vm at will, and also use this
> shared
> > ldap user database to allow for people to log on from computers/terminals
> > spread on campus. Home directories are basically pulled from the OS
> (mostly
> > ubuntu and centos) as a shared file system on the network, which then
> > synchronises changes live. Practically, that means a user will find its
> > desktop, preferences and files as they move to different computers.
> >
> > Theoretically, a user can connect to several computers at the same time,
> > which can lead to conflict issues at times, e.g. firefox doesn't like
> > parallel access to its preferences. I haven't fully tested it, but we
> think
> > that a guix profile is specific to a given computer, because of the way
> it
> > links to the store, which is (currently) local to the vm, which isn't
> > ideal: the whole point of using vms is for users to kill them when they
> > don't need them.
> >
> > What would be the recommended way of solving this? Has anybody had a
> > similar situation?
> > I am thinking we could either:
> > - leave it as is, and train users to recreate their profiles whenever
> they
> > use a new vm/computer (after all, that's what guix is for) but it could
> > take some time to recompile if binaries aren't available
> > - maybe turn /gnu/store as a shared nfs folder
>
> CCing guix-science
>
> One thing we do on the small cluster at UTenn is /gnu and /var/guix (in
> addition to the home directories) are exported across NFS and then any
> profiles that are installed in the users' home directories have their
> symlinks not dangling.
>
> I also have a snippet in my .profile to use GUIX_DAEMON_SOCKET to ssh
> back to the head node to use the guix-daemon there, which keeps all the
> guix stuff on one machine and easier to manage and make sure the same
> software is available on all the machines.
>
> --
> Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
> GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
> Confidentiality cannot be guaranteed on emails sent or received unencrypted
>

[-- Attachment #2: Type: text/html, Size: 3301 bytes --]

Re: guix on nfs based systems

[Felix Lechner via]

Hi Efraim,

On Thu, Nov 23 2023, Efraim Flashner wrote:

> One thing we do on the small cluster at UTenn is /gnu and /var/guix (in
> addition to the home directories) are exported across NFS

That's fascinating! Do you ever run 'guix gc', or do you just buy new
disks when space runs low?

Kind regards (and perhaps, happy Thanksgiving)
Felix

Re: guix on nfs based systems

[Felix Lechner via Guix-Science]

Hi Efraim,

On Thu, Nov 23 2023, Efraim Flashner wrote:

> One thing we do on the small cluster at UTenn is /gnu and /var/guix (in
> addition to the home directories) are exported across NFS

That's fascinating! Do you ever run 'guix gc', or do you just buy new
disks when space runs low?

Kind regards (and perhaps, happy Thanksgiving)
Felix

Re: guix on nfs based systems

[Etienne B. Roesch]

[-- Attachment #1: Type: text/plain, Size: 590 bytes --]

Hahaha! Yes, that's what I was getting at!

I suppose one could run gc periodically, in a monthly cron. No?

Etienne

On Thu, Nov 23, 2023 at 1:03 PM Felix Lechner <felix.lechner@lease-up.com>
wrote:

> Hi Efraim,
>
> On Thu, Nov 23 2023, Efraim Flashner wrote:
>
> > One thing we do on the small cluster at UTenn is /gnu and /var/guix (in
> > addition to the home directories) are exported across NFS
>
> That's fascinating! Do you ever run 'guix gc', or do you just buy new
> disks when space runs low?
>
> Kind regards (and perhaps, happy Thanksgiving)
> Felix
>

[-- Attachment #2: Type: text/html, Size: 1004 bytes --]

Re: guix on nfs based systems

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 3292 bytes --]

On Thu, Nov 23, 2023 at 09:41:56AM +0000, Etienne B. Roesch wrote:
> 
> On Thu, Nov 23, 2023 at 6:32 AM Efraim Flashner <efraim@flashner.co.il>
> wrote:
> 
> > On Wed, Nov 22, 2023 at 05:32:07PM +0000, Etienne B. Roesch wrote:
> > > Hi,
> > >
> > > I have been successful in convincing our IT dept to provide guix as
> > > standard on our vms and research clusters. We use this sophisticated
> > > platform that allows users to spawn a vm at will, and also use this
> > shared
> > > ldap user database to allow for people to log on from computers/terminals
> > > spread on campus. Home directories are basically pulled from the OS
> > (mostly
> > > ubuntu and centos) as a shared file system on the network, which then
> > > synchronises changes live. Practically, that means a user will find its
> > > desktop, preferences and files as they move to different computers.
> > >
> > > Theoretically, a user can connect to several computers at the same time,
> > > which can lead to conflict issues at times, e.g. firefox doesn't like
> > > parallel access to its preferences. I haven't fully tested it, but we
> > think
> > > that a guix profile is specific to a given computer, because of the way
> > it
> > > links to the store, which is (currently) local to the vm, which isn't
> > > ideal: the whole point of using vms is for users to kill them when they
> > > don't need them.
> > >
> > > What would be the recommended way of solving this? Has anybody had a
> > > similar situation?
> > > I am thinking we could either:
> > > - leave it as is, and train users to recreate their profiles whenever
> > they
> > > use a new vm/computer (after all, that's what guix is for) but it could
> > > take some time to recompile if binaries aren't available
> > > - maybe turn /gnu/store as a shared nfs folder
> >
> > CCing guix-science
> >
> > One thing we do on the small cluster at UTenn is /gnu and /var/guix (in
> > addition to the home directories) are exported across NFS and then any
> > profiles that are installed in the users' home directories have their
> > symlinks not dangling.
> >
> > I also have a snippet in my .profile to use GUIX_DAEMON_SOCKET to ssh
> > back to the head node to use the guix-daemon there, which keeps all the
> > guix stuff on one machine and easier to manage and make sure the same
> > software is available on all the machines.
> >
> Thanks! That's really helpful! How robust is your system to bandwidth
> fluctuations and network hiccups?
> How big is your store, /gnu, /var/guix?
> 
> How many users do you have?
> 
> Etienne

We have about a dozen users, with about 100GB for /gnu. It'd probably be
better with a larger store, but that's what we've partitioned for it.
/var/guix is rather small and not really worth worrying about.

We started with gigabit ethernet which seems to be fast enough for
accessing files from /gnu/store but could be faster when it comes to the
shared storage so we have a local scratch SSD on each system for local
files while doing computations.

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: guix on nfs based systems

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1188 bytes --]

On Fri, Nov 24, 2023 at 06:08:30PM +0000, Etienne B. Roesch wrote:
> Hahaha! Yes, that's what I was getting at!
> 
> I suppose one could run gc periodically, in a monthly cron. No?
> 
> Etienne
> 
> On Thu, Nov 23, 2023 at 1:03 PM Felix Lechner <felix.lechner@lease-up.com>
> wrote:
> 
> > Hi Efraim,
> >
> > On Thu, Nov 23 2023, Efraim Flashner wrote:
> >
> > > One thing we do on the small cluster at UTenn is /gnu and /var/guix (in
> > > addition to the home directories) are exported across NFS
> >
> > That's fascinating! Do you ever run 'guix gc', or do you just buy new
> > disks when space runs low?
> >
> > Kind regards (and perhaps, happy Thanksgiving)
> > Felix

We generally run it as necessary, but luckily not too often.  With
everyone trained to use profiles we rarely have problems with necessary
packages disappearing.  I've generally been the biggest offender with
building and rebuilding packages while packaging new software.

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: guix on nfs based systems

[Etienne B. Roesch]

Hi!

A quick follow up on that, as I have had some discussions with a few of
you, and also my IT department.

*Context*: at the University of Reading, we have that facility that allows
the deployment of vms on-demand to the campus. I work primarily on
reproducibility in science, and managed to convince our IT dept to provide
guix as default to all users.

*Hiccups*: we provide home dirs as nfs drives through the network. Using
guix, we are thinking of creating one nfs drive, shared by all users, to
contain /var/guix and /gnu/store, symlinked from /.
As I understand, that should work, until a user decides to run "guix gc"
(which would clear wrongly assumed unused profiles) or maybe until a user
decides to launch several vms (which is theoretically possible, but doesn't
happen often).

Efraim suggested using a shared daemon ssh-ing GUIX_DAEMON_SOCKET. We would
probably run this on a separate vm. We are however unsure how it would
behave when /var/guix/daemon-socket/socket is itself on an nfs.

In theory, it should work, right? Are we missing something?

Thanks a ton!

Etienne



On Sun, Nov 26, 2023 at 7:36 AM Efraim Flashner <efraim@flashner.co.il>
wrote:

> On Fri, Nov 24, 2023 at 06:08:30PM +0000, Etienne B. Roesch wrote:
> > Hahaha! Yes, that's what I was getting at!
> >
> > I suppose one could run gc periodically, in a monthly cron. No?
> >
> > Etienne
> >
> > On Thu, Nov 23, 2023 at 1:03 PM Felix Lechner <
> felix.lechner@lease-up.com>
> > wrote:
> >
> > > Hi Efraim,
> > >
> > > On Thu, Nov 23 2023, Efraim Flashner wrote:
> > >
> > > > One thing we do on the small cluster at UTenn is /gnu and /var/guix
> (in
> > > > addition to the home directories) are exported across NFS
> > >
> > > That's fascinating! Do you ever run 'guix gc', or do you just buy new
> > > disks when space runs low?
> > >
> > > Kind regards (and perhaps, happy Thanksgiving)
> > > Felix
>
> We generally run it as necessary, but luckily not too often.  With
> everyone trained to use profiles we rarely have problems with necessary
> packages disappearing.  I've generally been the biggest offender with
> building and rebuilding packages while packaging new software.
>
> --
> Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
> GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
> Confidentiality cannot be guaranteed on emails sent or received unencrypted
>

Re: guix on nfs based systems

[Ricardo Wurmus]

Hi Etienne,

FWIW at the MDC we’ve been doing the same: on all nodes mount /gnu read-only, mount
/var/guix read-write; provide a “guix” executable that sets
GUIX_DAEMON_SOCKET and disables/discourages “guix gc”.

> Thanks! That's really helpful! How robust is your system to bandwidth
> fluctuations and network hiccups?

Network reliability has never been an issue for us when it comes to
Guix.  We serve /gnu redundantly with pacemaker so that we can take down
one of the two Guix servers (they are VMs) without affecting
availibility of programs and files on /gnu.

> How big is your store, /gnu, /var/guix?

We have about 2TB for /gnu.  We never run “guix gc” because the Guix
servers have no access to all the users’ shares, so it would be overly
aggressive in determining garbage.

> How many users do you have?

About 200.

-- 
Ricardo Wurmus

System administrator
BIMSB - Scientific Bioinformatics Platform
Max Delbrueck Center for Molecular Medicine

email: ricardo.wurmus@mdc-berlin.de
tel:   +49 30 9406 1796

Re: guix on nfs based systems

[Ricardo Wurmus]

"Etienne B. Roesch" <etienne.roesch@gmail.com> writes:

> Hiccups: we provide home dirs as nfs drives through the network. Using guix, we are thinking of creating one nfs drive, shared by all
> users, to contain /var/guix and /gnu/store, symlinked from /.
> As I understand, that should work, until a user decides to run "guix gc" (which would clear wrongly assumed unused profiles) or maybe
> until a user decides to launch several vms (which is theoretically possible, but doesn't happen often).

I would strongly discourage the use of profiles in users’ home
directories.  When introducing Guix we now only demo “guix shell”, which
is preferrable in most cases.  “guix gc” is problematic when profile
links are in locations that the daemon cannot read. 

> Efraim suggested using a shared daemon ssh-ing GUIX_DAEMON_SOCKET. We
> would probably run this on a separate vm. We are however unsure how it
> would behave when /var/guix/daemon-socket/socket is itself on an nfs.

We only export /var/guix/profiles, not anything else in /var/guix.
Using GUIX_DAEMON_SOCKET with a network port (make “guix-daemon” listen
on that port) is all we ever needed.

-- 
Ricardo Wurmus

System administrator
BIMSB - Scientific Bioinformatics Platform
Max Delbrueck Center for Molecular Medicine

email: ricardo.wurmus@mdc-berlin.de
tel:   +49 30 9406 1796

Re: guix on nfs based systems

[Etienne B. Roesch]

[-- Attachment #1: Type: text/plain, Size: 2653 bytes --]

Thanks a ton!

I think we've arrived at the limit of how I understand the daemon to work,
and GUIX_DAEMON_SOCKET. I think I understand that you are using a single
node (hpc of sort I imagine), where users create sessions, and within which
you provide the guix command, having set up GUIX_DAEMON_SOCKET to a
unix-domain socket (to that same node / itself). That makes total sense in
the context of the single node. Did I get that right?

I think what we are aiming for, in our case (where users each have their
own nodes as it were, only sharing network drives), is providing the guix
command on each node, set up with GUIX_DAEMON_SOCKET connecting with ssh to
a master node with a daemon, that itself would have access to the same
network drives.

The issue with profiles you are mentioning is interesting; I haven't quite
thought it through yet. I think I would personally want users to be able to
create profiles (for reproducibility reasons) but I guess it would work the
same way with guix shells built from manifests, maybe slightly less easy to
interact with, I don't know.

Etienne

On Thu, Dec 14, 2023 at 2:48 PM Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>
wrote:

>
> "Etienne B. Roesch" <etienne.roesch@gmail.com> writes:
>
> > Hiccups: we provide home dirs as nfs drives through the network. Using
> guix, we are thinking of creating one nfs drive, shared by all
> > users, to contain /var/guix and /gnu/store, symlinked from /.
> > As I understand, that should work, until a user decides to run "guix gc"
> (which would clear wrongly assumed unused profiles) or maybe
> > until a user decides to launch several vms (which is theoretically
> possible, but doesn't happen often).
>
> I would strongly discourage the use of profiles in users’ home
> directories.  When introducing Guix we now only demo “guix shell”, which
> is preferrable in most cases.  “guix gc” is problematic when profile
> links are in locations that the daemon cannot read.
>
> > Efraim suggested using a shared daemon ssh-ing GUIX_DAEMON_SOCKET. We
> > would probably run this on a separate vm. We are however unsure how it
> > would behave when /var/guix/daemon-socket/socket is itself on an nfs.
>
> We only export /var/guix/profiles, not anything else in /var/guix.
> Using GUIX_DAEMON_SOCKET with a network port (make “guix-daemon” listen
> on that port) is all we ever needed.
>
> --
> Ricardo Wurmus
>
> System administrator
> BIMSB - Scientific Bioinformatics Platform
> Max Delbrueck Center for Molecular Medicine
>
> email: ricardo.wurmus@mdc-berlin.de
> tel:   +49 30 9406 1796
>

[-- Attachment #2: Type: text/html, Size: 3308 bytes --]

Re: guix on nfs based systems

[Pierre-Antoine Bouttier]

[-- Attachment #1: Type: text/plain, Size: 3352 bytes --]

Hi Etienne, 

> The issue with profiles you are mentioning is interesting; I haven't quite
> thought it through yet. I think I would personally want users to be able to
> create profiles (for reproducibility reasons) but I guess it would work the
> same way with guix shells built from manifests, maybe slightly less easy to
> interact with, I don't know.

As part of the support team in a HPC center that provides guix command to our users to set up their software environments, we advice them to use guix shell rather than guix profile and most of them are pretty happy with that. 
Obviously, you need to write a good documentation :)

My 2 cents
P-A
---
Pierre-Antoine Bouttier
CNRS Research Engineer
Dir. Adj. UAR GRICAD

GriCAD - https://gricad.univ-grenoble-alpes.fr/
Batiment IMAG
CS 40700
38058 Grenoble CEDEX 9

+33 4 57 42 18  66

> Le 14 déc. 2023 à 16:28, Etienne B. Roesch <etienne.roesch@gmail.com> a écrit :
> 
> Thanks a ton!
> 
> I think we've arrived at the limit of how I understand the daemon to work,
> and GUIX_DAEMON_SOCKET. I think I understand that you are using a single
> node (hpc of sort I imagine), where users create sessions, and within which
> you provide the guix command, having set up GUIX_DAEMON_SOCKET to a
> unix-domain socket (to that same node / itself). That makes total sense in
> the context of the single node. Did I get that right?
> 
> I think what we are aiming for, in our case (where users each have their
> own nodes as it were, only sharing network drives), is providing the guix
> command on each node, set up with GUIX_DAEMON_SOCKET connecting with ssh to
> a master node with a daemon, that itself would have access to the same
> network drives.
> 
> 
> Etienne
> 
> On Thu, Dec 14, 2023 at 2:48 PM Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>
> wrote:
> 
>> 
>> "Etienne B. Roesch" <etienne.roesch@gmail.com> writes:
>> 
>>> Hiccups: we provide home dirs as nfs drives through the network. Using
>> guix, we are thinking of creating one nfs drive, shared by all
>>> users, to contain /var/guix and /gnu/store, symlinked from /.
>>> As I understand, that should work, until a user decides to run "guix gc"
>> (which would clear wrongly assumed unused profiles) or maybe
>>> until a user decides to launch several vms (which is theoretically
>> possible, but doesn't happen often).
>> 
>> I would strongly discourage the use of profiles in users’ home
>> directories.  When introducing Guix we now only demo “guix shell”, which
>> is preferrable in most cases.  “guix gc” is problematic when profile
>> links are in locations that the daemon cannot read.
>> 
>>> Efraim suggested using a shared daemon ssh-ing GUIX_DAEMON_SOCKET. We
>>> would probably run this on a separate vm. We are however unsure how it
>>> would behave when /var/guix/daemon-socket/socket is itself on an nfs.
>> 
>> We only export /var/guix/profiles, not anything else in /var/guix.
>> Using GUIX_DAEMON_SOCKET with a network port (make “guix-daemon” listen
>> on that port) is all we ever needed.
>> 
>> --
>> Ricardo Wurmus
>> 
>> System administrator
>> BIMSB - Scientific Bioinformatics Platform
>> Max Delbrueck Center for Molecular Medicine
>> 
>> email: ricardo.wurmus@mdc-berlin.de
>> tel:   +49 30 9406 1796
>> 


[-- Attachment #2: Type: text/html, Size: 5455 bytes --]

Re: guix on nfs based systems

[Pierre-Antoine Bouttier]

Hi Etienne, 

> The issue with profiles you are mentioning is interesting; I haven't quite
> thought it through yet. I think I would personally want users to be able to
> create profiles (for reproducibility reasons) but I guess it would work the
> same way with guix shells built from manifests, maybe slightly less easy to
> interact with, I don't know.

As part of the support team in a HPC center that provides guix command to our users to set up their software environments, we advice them to use guix shell rather than guix profile and most of them are pretty happy with that. 
Obviously, you need to write a good documentation :)

My 2 cents
P-A
---
Pierre-Antoine Bouttier
CNRS Research Engineer
Dir. Adj. UAR GRICAD

GriCAD - https://gricad.univ-grenoble-alpes.fr/
Batiment IMAG
CS 40700
38058 Grenoble CEDEX 9

+33 4 57 42 18  66

> Le 14 déc. 2023 à 16:28, Etienne B. Roesch <etienne.roesch@gmail.com> a écrit :
> 
> Thanks a ton!
> 
> I think we've arrived at the limit of how I understand the daemon to work,
> and GUIX_DAEMON_SOCKET. I think I understand that you are using a single
> node (hpc of sort I imagine), where users create sessions, and within which
> you provide the guix command, having set up GUIX_DAEMON_SOCKET to a
> unix-domain socket (to that same node / itself). That makes total sense in
> the context of the single node. Did I get that right?
> 
> I think what we are aiming for, in our case (where users each have their
> own nodes as it were, only sharing network drives), is providing the guix
> command on each node, set up with GUIX_DAEMON_SOCKET connecting with ssh to
> a master node with a daemon, that itself would have access to the same
> network drives.
> 
> 
> Etienne
> 
> On Thu, Dec 14, 2023 at 2:48 PM Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>
> wrote:
> 
>> 
>> "Etienne B. Roesch" <etienne.roesch@gmail.com> writes:
>> 
>>> Hiccups: we provide home dirs as nfs drives through the network. Using
>> guix, we are thinking of creating one nfs drive, shared by all
>>> users, to contain /var/guix and /gnu/store, symlinked from /.
>>> As I understand, that should work, until a user decides to run "guix gc"
>> (which would clear wrongly assumed unused profiles) or maybe
>>> until a user decides to launch several vms (which is theoretically
>> possible, but doesn't happen often).
>> 
>> I would strongly discourage the use of profiles in users’ home
>> directories.  When introducing Guix we now only demo “guix shell”, which
>> is preferrable in most cases.  “guix gc” is problematic when profile
>> links are in locations that the daemon cannot read.
>> 
>>> Efraim suggested using a shared daemon ssh-ing GUIX_DAEMON_SOCKET. We
>>> would probably run this on a separate vm. We are however unsure how it
>>> would behave when /var/guix/daemon-socket/socket is itself on an nfs.
>> 
>> We only export /var/guix/profiles, not anything else in /var/guix.
>> Using GUIX_DAEMON_SOCKET with a network port (make “guix-daemon” listen
>> on that port) is all we ever needed.
>> 
>> --
>> Ricardo Wurmus
>> 
>> System administrator
>> BIMSB - Scientific Bioinformatics Platform
>> Max Delbrueck Center for Molecular Medicine
>> 
>> email: ricardo.wurmus@mdc-berlin.de
>> tel:   +49 30 9406 1796
>> 

Re: guix on nfs based systems

[Etienne B. Roesch]

[-- Attachment #1: Type: text/plain, Size: 4962 bytes --]

Oops, my bad, I seemed to have hit Reply, instead of Reply-all.

Here is the follow up to that email:


*Etienne B. Roesch <etienne.roesch@gmail.com <etienne.roesch@gmail.com>>
3:42 PM (12 minutes ago) to Pierre-Antoine*
Merci Pierre-Antoine ! :)

It sounds like you also made the conscious choice of using guix shells,
instead of profiles. Why is that? I am guessing profiles have a more
widespread impact on what's available to the user, modifying paths and so
on, whereas shells would be more contained and short-lived, and therefore
"safer" for users?

I'll have to write documentation and train users either way ;)

Etienne


*Pierre-Antoine Bouttier 3:47 PM (7 minutes ago) to me*

> Merci Pierre-Antoine ! :)

My pleasure :)

> It sounds like you also made the conscious choice of using guix shells,
instead of profiles.
> Why is that?

Because

> I am guessing profiles have a more widespread impact on what's available
to the user,
> modifying paths and so on, whereas shells would be more contained and
short-lived, and
> therefore "safer" for users?

;)

Yes, indeed, the isolated and self-contained aspect of guix shell avoid to
mess with environment variables. And it’s far more easier, in a
reproducibility point of view, with manifest.scm and channels.scm to
explain to users how to work with guix time-machine and guix shell.

But, our users can use guix profile (and some of them don’t hold back).

P-A

On Thu, Dec 14, 2023 at 3:33 PM Pierre-Antoine Bouttier <
Pierre-Antoine.Bouttier@univ-grenoble-alpes.fr> wrote:

> Hi Etienne,
>
> The issue with profiles you are mentioning is interesting; I haven't quite
> thought it through yet. I think I would personally want users to be able to
> create profiles (for reproducibility reasons) but I guess it would work the
> same way with guix shells built from manifests, maybe slightly less easy to
> interact with, I don't know.
>
>
> As part of the support team in a HPC center that provides guix command to
> our users to set up their software environments, we advice them to use guix
> shell rather than guix profile and most of them are pretty happy with that.
> Obviously, you need to write a good documentation :)
>
> My 2 cents
> P-A
> ---
> Pierre-Antoine Bouttier
> CNRS Research Engineer
> Dir. Adj. UAR GRICAD
>
> GriCAD - https://gricad.univ-grenoble-alpes.fr/
> Batiment IMAG
> CS 40700
> 38058 Grenoble CEDEX 9
>
> +33 4 57 42 18  66
>
> Le 14 déc. 2023 à 16:28, Etienne B. Roesch <etienne.roesch@gmail.com> a
> écrit :
>
> Thanks a ton!
>
> I think we've arrived at the limit of how I understand the daemon to work,
> and GUIX_DAEMON_SOCKET. I think I understand that you are using a single
> node (hpc of sort I imagine), where users create sessions, and within which
> you provide the guix command, having set up GUIX_DAEMON_SOCKET to a
> unix-domain socket (to that same node / itself). That makes total sense in
> the context of the single node. Did I get that right?
>
> I think what we are aiming for, in our case (where users each have their
> own nodes as it were, only sharing network drives), is providing the guix
> command on each node, set up with GUIX_DAEMON_SOCKET connecting with ssh to
> a master node with a daemon, that itself would have access to the same
> network drives.
>
>
> Etienne
>
> On Thu, Dec 14, 2023 at 2:48 PM Ricardo Wurmus <
> ricardo.wurmus@mdc-berlin.de>
> wrote:
>
>
> "Etienne B. Roesch" <etienne.roesch@gmail.com> writes:
>
> Hiccups: we provide home dirs as nfs drives through the network. Using
>
> guix, we are thinking of creating one nfs drive, shared by all
>
> users, to contain /var/guix and /gnu/store, symlinked from /.
> As I understand, that should work, until a user decides to run "guix gc"
>
> (which would clear wrongly assumed unused profiles) or maybe
>
> until a user decides to launch several vms (which is theoretically
>
> possible, but doesn't happen often).
>
> I would strongly discourage the use of profiles in users’ home
> directories.  When introducing Guix we now only demo “guix shell”, which
> is preferrable in most cases.  “guix gc” is problematic when profile
> links are in locations that the daemon cannot read.
>
> Efraim suggested using a shared daemon ssh-ing GUIX_DAEMON_SOCKET. We
> would probably run this on a separate vm. We are however unsure how it
> would behave when /var/guix/daemon-socket/socket is itself on an nfs.
>
>
> We only export /var/guix/profiles, not anything else in /var/guix.
> Using GUIX_DAEMON_SOCKET with a network port (make “guix-daemon” listen
> on that port) is all we ever needed.
>
> --
> Ricardo Wurmus
>
> System administrator
> BIMSB - Scientific Bioinformatics Platform
> Max Delbrueck Center for Molecular Medicine
>
> email: ricardo.wurmus@mdc-berlin.de
> tel:   +49 30 9406 1796
>
>
>

[-- Attachment #2: Type: text/html, Size: 7291 bytes --]

Re: [ext] Re: guix on nfs based systems

[Ricardo Wurmus]

Hi Etienne,

> I think we've arrived at the limit of how I understand the daemon to work, and GUIX_DAEMON_SOCKET. I think I understand that you
> are using a single node (hpc of sort I imagine), where users create sessions, and within which you provide the guix command, having
> set up GUIX_DAEMON_SOCKET to a unix-domain socket (to that same node / itself). That makes total sense in the context of the single
> node. Did I get that right?

No, we provide Guix on *all* cluster nodes and workstations, but they
all talk to a single guix-daemon, which is the sole manager of /gnu.  We
provide a default Guix for all those who haven’t yet run “guix pull” at
/gnu/remote/bin/guix.  That location is on the PATH on all machines.

This script does three things:

- it sets GUIX_DAEMON_SOCKET to http://guix-node:9999
- it removes “gc” from the arguments and prints a warning (this is not
  enough to prevent GC, but it works well enough)
- it looks for the user’s “guix” in /home/who/.config/guix/current/bin;
  if it exists it execs that “guix” with the arguments; otherwise it
  executes the shared default Guix.

> The issue with profiles you are mentioning is interesting; I haven't
> quite thought it through yet. I think I would personally want users to
> be able to create profiles (for reproducibility reasons) but I guess
> it would work the same way with guix shells built from manifests,
> maybe slightly less easy to interact with, I don't know.

In my experience, profiles always end up growing in a stateful manner.
People use them to satisfy old habits of installing, removing, and
upgrading packages, and inevitably they run into conflicts because they
install new packages to a profile built with an older version of Guix.
Yes, using manifests would avoid that, but if you’re already using
manifests you might as well use “guix shell -m”.  For long-term
reproducibility we recommend recording the current channels with “guix
describe -f channels” and add the output to version control.

With “guix time-machine -C channels.scm -- shell -m manifest.scm” people
can restore the environment in the future.

-- 
Ricardo Wurmus

System administrator
BIMSB - Scientific Bioinformatics Platform
Max Delbrueck Center for Molecular Medicine

email: ricardo.wurmus@mdc-berlin.de
tel:   +49 30 9406 1796

Re: guix on nfs based systems

[Simon Tournier]

Hi Étienne,

I am very late to the party. :-)

Well, there is a discussion about “guix shell”, “profile” and “guix
time-machine”.  Since these concepts can appear as first as alien, let
me try a quick summary of my understanding.

 + “guix shell“ creates a temporary profile;
 + “guix time-machine” creates a temporary revision.

For the plumbing details, both live under “~/.cache/guix/” and all is
managed automatically.

An example:

    $ guix package -m manifest.smc -p /tmp/foo
    $ GUIX_PROFILE="/tmp/foo"
    $ . "$GUIX_PROFILE/etc/profile"

Instead,

    $ guix shell -m manifest.scm

somehow does the same but the user does not manually manage the profile
and activate it.  This profile is automatically and transparently
managed.

Now, you might run “guix pull”, so there is a chance that the next “guix
shell” will not build the same underlying profile.  That’s where “guix
time-machine” comes in. :-)

You “freeze” one revision with “guix describe -f channels >
channels.scm” then run

    guix time-machine -C channels.scm -- shell manifest.scm

It is costly the first run, then all is cached.  Somehow, this command
could be seen as “conda env activate“ coupled to some lock stuff.

All that said, one thing to know: when is the cache cleaned up?

There is a ’last-expiry-cleanup’ that tracks and does the job.
Basically, you do not have to worry about this ~/.cache/guix/.  All is
automatically managed.

When the user runs ‘guix gc’ or the sysadmin also ’guix gc’ then this
cache is cleaned up if it is too old.

Now the question is about what does it mean “too old”?  Although “guix
time-machine -C channels.scm -- shell -m manifest.scm” will rebuild all
if required, it can be at the wrong moment.  Therefore, for some very
specific environment that I want to protect from “guix gc”, I create a
profile for them.  Note it is rare and thanks many improvements, it
becomes more and more useless.

Obviously, if “guix gc” is never run, the manual management of “profile”
is fully useless, IMHO.


Cheers,
simon