guix on a cluster: install into existing mount

[Adam McCartney via Guix-Science <guix-science@gnu.org>]

Dear Guix-Science!

Our team at the HPC center in Vienna is currently exploring possible options for
a tool to manage our software stacks going forward. I'm quite happy to have the
job of setting up a guix cluster to do some testing!

If I understand the "Setting Up a Head Node" section of the tutorial correctly
the idea is to make the head node an nfs server that provides the `/gnu/store`,
`/var/guix` and `/var/log/guix`.[^1]

The approach that I'm trying differs slightly in that I'd like to use an
existing nfs server to handle the job of providing the store both to the head
nodes and to other nodes on the cluster. I've already found some useful
information in previous threads.[^2][^3] But wanted to ask here for some advice
from you all on the specifics of what I am trying to do.

As I mentioned, we're already using gpfs to provide a `/mnt/sw` on all the nodes
in the cluster. For test purposes, I'm setting up a directory tree on the file
system, after mounting it looks like this:

```
/mnt/sw/dev/guix/
├── gnu
│   └── store
└── var
     ├── guix
     └── log
         └── guix
```

Using `mount --rbind`, I can remount the parts of the hierarchy that I need to
their expected locations (`/gnu/store`, `/var/guix` and `/var/log/guix`).

I then take a very hacky swipe at the `guix-install.sh` by commenting out the
checks for existing "/gnu/store" and "/var/guix", then running the install
script. The install process stalled for a while at the step
`[ INFO ] Installing /var/guix and /gnu...`, but then went on the complete.

Much to my surprise and delight, this appears to have worked to some degree.
At least the store seems to have been populated by the initial install.

There are a couple of things that still need to be ironed out:
+ The recursively mounted subtrees don't survive a reboot, despite there being
   fstab entries for them.
+ The guix daemon doesn't want to run, I think this maybe has something to do
   with the way that the `gnu.mount` service is set up? I guess that unit is
   maybe not expecting to find a mount under `/gnu/store`?

Has anyone had similar experiences and/or could possibly spot a few places where
I'm going wrong?

Regards,
Adam

[^1]: <https://guix.gnu.org/cookbook/en/html_node/Setting-Up-a-Head-Node.html>
[^2]: <https://lists.gnu.org/archive/html/guix-science/2021-03/msg00000.html>
[^3]: <https://lists.gnu.org/archive/html/guix-science/2023-11/msg00001.html>


-- 
  Adam McCartney - https://admccartney.mur.at 
/