guix on a cluster: install into existing mount

guix on a cluster: install into existing mount

[Adam McCartney via Guix-Science]

Dear Guix-Science!

Our team at the HPC center in Vienna is currently exploring possible options for
a tool to manage our software stacks going forward. I'm quite happy to have the
job of setting up a guix cluster to do some testing!

If I understand the "Setting Up a Head Node" section of the tutorial correctly
the idea is to make the head node an nfs server that provides the `/gnu/store`,
`/var/guix` and `/var/log/guix`.[^1]

The approach that I'm trying differs slightly in that I'd like to use an
existing nfs server to handle the job of providing the store both to the head
nodes and to other nodes on the cluster. I've already found some useful
information in previous threads.[^2][^3] But wanted to ask here for some advice
from you all on the specifics of what I am trying to do.

As I mentioned, we're already using gpfs to provide a `/mnt/sw` on all the nodes
in the cluster. For test purposes, I'm setting up a directory tree on the file
system, after mounting it looks like this:

```
/mnt/sw/dev/guix/
├── gnu
│   └── store
└── var
     ├── guix
     └── log
         └── guix
```

Using `mount --rbind`, I can remount the parts of the hierarchy that I need to
their expected locations (`/gnu/store`, `/var/guix` and `/var/log/guix`).

I then take a very hacky swipe at the `guix-install.sh` by commenting out the
checks for existing "/gnu/store" and "/var/guix", then running the install
script. The install process stalled for a while at the step
`[ INFO ] Installing /var/guix and /gnu...`, but then went on the complete.

Much to my surprise and delight, this appears to have worked to some degree.
At least the store seems to have been populated by the initial install.

There are a couple of things that still need to be ironed out:
+ The recursively mounted subtrees don't survive a reboot, despite there being
   fstab entries for them.
+ The guix daemon doesn't want to run, I think this maybe has something to do
   with the way that the `gnu.mount` service is set up? I guess that unit is
   maybe not expecting to find a mount under `/gnu/store`?

Has anyone had similar experiences and/or could possibly spot a few places where
I'm going wrong?

Regards,
Adam

[^1]: <https://guix.gnu.org/cookbook/en/html_node/Setting-Up-a-Head-Node.html>
[^2]: <https://lists.gnu.org/archive/html/guix-science/2021-03/msg00000.html>
[^3]: <https://lists.gnu.org/archive/html/guix-science/2023-11/msg00001.html>


-- 
  Adam McCartney - https://admccartney.mur.at 
/

Re: guix on a cluster: install into existing mount

[Adam McCartney via Guix-Science]

Can also offer a little bit of further clarification:

>+ The recursively mounted subtrees don't survive a reboot, despite there being
>  fstab entries for them.
>+ The guix daemon doesn't want to run, I think this maybe has something to do
>  with the way that the `gnu.mount` service is set up? I guess that unit is
>  maybe not expecting to find a mount under `/gnu/store`?

So, the typical pattern that was being applied when I approached the system was 
append the following two entries to fstab:

```
[HOST_IP]:/gpfs/opt/adm /mnt/adm nfs rw,async,noatime,nodiratime,_netdev 0 0
[HOST_IP]:/gpfs/opt/sw /mnt/sw nfs rw,async,noatime,nodiratime,_netdev 0 0
```

As I mentioned in the thread above, I've recreated the required guix folders
as:
  + `/mnt/sw/dev/guix/gnu/store`
  + `/mnt/sw/dev/guix/var/guix`
  + `/mnt/sw/dev/guix/var/log/guix`

And then used the recursive bind option for mount:

```
mount --rbind /mnt/sw/dev/guix/gnu/store/ /gnu/store
mount --rbind /mnt/sw/dev/guix/var/guix/ /var/guix
mount --rbind /mng/sw/dev/guix/var/log/guix/ /var/log/guix
```

At this point it's possible to start the guix daemon and do a guix pull.

I'm a bit unsure as to how to make these recursive bind mounts persistent in
fstab. I've tried a couple of different options:

As type nfs rbind
```
/mnt/sw/dev/guix/gnu/store /gnu/store nfs rbind
...
```

As type none rbind
```
/mnt/sw/dev/guix/gnu/store /gnu/store none rbind
...
```

Referencing the host directly
```
[HOST_IP]:/gpfs/sw/dev/guix/gnu/store /gnu/store nfs rbind
...
```

Any pointers in the direction of where I might find some more info on making 
recursive bind mounts persistent much appreciated!

cheers,
Adam

p.s. I have considered that a cleaner solution would be to actually export the 
relevant directories from the nfs server rather than creating bind mounts after
it has been mounted. This is probably an option going forward, but for the time
being I probably have to make do with the test setup as described above.



-- 
  Adam McCartney - https://admccartney.mur.at 
/

Re: guix on a cluster: install into existing mount

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2814 bytes --]

On Thu, Jul 04, 2024 at 08:40:54PM +0200, Adam McCartney via Guix-Science wrote:
> Can also offer a little bit of further clarification:
> 
> > + The recursively mounted subtrees don't survive a reboot, despite there being
> >  fstab entries for them.
> > + The guix daemon doesn't want to run, I think this maybe has something to do
> >  with the way that the `gnu.mount` service is set up? I guess that unit is
> >  maybe not expecting to find a mount under `/gnu/store`?
> 
> So, the typical pattern that was being applied when I approached the system
> was append the following two entries to fstab:
> 
> ```
> [HOST_IP]:/gpfs/opt/adm /mnt/adm nfs rw,async,noatime,nodiratime,_netdev 0 0
> [HOST_IP]:/gpfs/opt/sw /mnt/sw nfs rw,async,noatime,nodiratime,_netdev 0 0
> ```
> 
> As I mentioned in the thread above, I've recreated the required guix folders
> as:
>  + `/mnt/sw/dev/guix/gnu/store`
>  + `/mnt/sw/dev/guix/var/guix`
>  + `/mnt/sw/dev/guix/var/log/guix`
> 
> And then used the recursive bind option for mount:
> 
> ```
> mount --rbind /mnt/sw/dev/guix/gnu/store/ /gnu/store
> mount --rbind /mnt/sw/dev/guix/var/guix/ /var/guix
> mount --rbind /mng/sw/dev/guix/var/log/guix/ /var/log/guix
> ```
> 
> At this point it's possible to start the guix daemon and do a guix pull.
> 
> I'm a bit unsure as to how to make these recursive bind mounts persistent in
> fstab. I've tried a couple of different options:
> 
> As type nfs rbind
> ```
> /mnt/sw/dev/guix/gnu/store /gnu/store nfs rbind
> ...
> ```
> 
> As type none rbind
> ```
> /mnt/sw/dev/guix/gnu/store /gnu/store none rbind
> ...
> ```
> 
> Referencing the host directly
> ```
> [HOST_IP]:/gpfs/sw/dev/guix/gnu/store /gnu/store nfs rbind
> ...
> ```
> 
> Any pointers in the direction of where I might find some more info on making
> recursive bind mounts persistent much appreciated!
> 
> cheers,
> Adam
> 
> p.s. I have considered that a cleaner solution would be to actually export
> the relevant directories from the nfs server rather than creating bind
> mounts after
> it has been mounted. This is probably an option going forward, but for the time
> being I probably have to make do with the test setup as described above.
> 

It could be that there isn't a /gnu directory already on the machine, so
could it be that's why it's not working?

I guess my two questions would be:
1. Do any of the mounts work? (that is, can you test them individually
instead of all at once?)
2. Does something like the following work?
/mnt/sw/dev/guix/gnu /gnu none rbind

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: guix on a cluster: install into existing mount

[Adam McCartney via Guix-Science]

Hi Efraim,

>It could be that there isn't a /gnu directory already on the machine, so
>could it be that's why it's not working?
>
>I guess my two questions would be:
>1. Do any of the mounts work? (that is, can you test them individually
>instead of all at once?)

Thanks for the clarification - yes, the issue was that I was trying to mount 
`store` as without there being a `/gnu` directory present.

>2. Does something like the following work?
>/mnt/sw/dev/guix/gnu /gnu none rbind

Yes indeed, this works.

The others:
/mnt/sw/dev/guix/var/guix /var/guix none rbind
/mnt/sw/dev/guix/var/log/guix /var/log/guix none rbind 

Are also working fine (in fact they may have been fine to begin with!)

Thanks for your help.

-- 
  Adam McCartney - https://admccartney.mur.at 
/