Re: Guix on clusters and in HPC

[ludo@gnu.org (Ludovic Courtès)]

Hi Roel,

Roel Janssen <roel@gnu.org> skribis:

> Here are some aspects I think we need:
>
> * Network-aware guix-daemon

Of course!

> * Profile management
>
>   The abstraction of profiles is an awesome feature of FPM, but the user
>   interface is missing.  We could do better here.
>
>   Switch the default profile
>   (and prepend values of environment variables to the current values):
>   $ guix profile --switch=/path/to/shared/profile
>
>   Reset to default profile (and environment variable values without the
>   profile we just unset):
>   $ guix profile --reset
>
>   Create an isolated environment based on a profile:
>   $ guix environment --profile=/path/to/profile --pure --ad-hoc

I can see the desire of having something that more closely resembles
what “modules” does, but I have the same questions as Ricardo.
Essentially, how much would it help compared to what’s already
available?  (Honest question.)

In general, adding simpler UIs is a good idea IMO; it’s just that I’m
unsure what’s “better” in this case.

> * Workflow management/execution
>
>   Add automatic program execution with its own vocabulary.  I think
>   "workflow management" boils down to execution of a G-exp, but the
>   results do not necessarily need to be stored in the store (because the
>   data it works on is probably managed by an external data management
>   system).  A powerful feature of GNU Guix is its domain-specific
>   language for describing software packages.  We could add
>   domain-specific parts for workflow management (a `workflow' data type
>   and a `task' or `process' data type gets us there more or less).
>
>   With workflow management we are only interested in the "build
>   function", not the "source code" or the "build output".
>
>   You are probably aware that I worked on this for some time, so I could
>   share the data types I have and the execution engine parts I have.

Yes, definitely!  This is what I had in mind, hence the reference to
<https://lists.gnu.org/archive/html/guix-devel/2016-05/msg00380.html>.

Obviously if there’s already code, it’s even better.  :-)

>   The HPC-specific part of this is the compatibility with existing job
>   scheduling systems and data management systems.

Do you mean that it integrates with a job scheduler?

> * Document on why we need super user privileges on the Guix daemon
>
>   Probably an infamous point by now.  By design, the Linux kernel keeps
>   control over all processes.  With GNU Guix, we need some control over
>   the environment in which a process runs (disable network access,
>   change the user that executes a process), and the environment in which
>   the output lives (chown, chmod, to allow multiple users to use the
>   build output).  Instead of hitting the wall of "we are not going to
>   run this thing with root privileges", we could present our sysadmins
>   with a document for the reasons, the design decisions and the actual
>   code involved in super user privilege stuff.
>
>   This is something I am working on as well, but help is always welcome
>   :-).

Good point.
<https://www.gnu.org/software/guix/manual/html_node/Build-Environment-Setup.html>
mentions it when talking about --disable-chroot at the end, but this
could be improved.

That’s it?  No crazy ideas?  ;-)

Your thoughts about the point about Galaxy?

Thanks for your feedback!

Ludo’.