Introducing Guix to HPC at my institution

Introducing Guix to HPC at my institution

[Sébastien Lerique]

Hi all,

I'm interested in introducing Guix to the HPC team at my 
institution. There's quite some content on the Guix-HPC blog [0], 
but I was wondering if people here have stories about how such 
conversations went for them, which arguments worked best, which 
didn't.

As a first step I would like to discuss with the sysadmins if they 
would consider activating user namespaces so users can play 
around. Aside from [1] and [2], are there any updated discussions 
about this topic, e.g. why to do it or why not?

Best,
Sébastien


[0] https://hpc.guix.info/blog/
[1] 
https://hpc.guix.info/blog/2017/10/using-guix-without-being-root/
[2] 
https://hpc.guix.info/blog/2017/09/reproducibility-and-root-privileges/

Re: Introducing Guix to HPC at my institution

[zimoun]

Hi Sébastien,

On Mon, 15 Mar 2021 at 06:19, Sébastien Lerique <sl@eauchat.org> wrote:

> I'm interested in introducing Guix to the HPC team at my
> institution. There's quite some content on the Guix-HPC blog [0],
> but I was wondering if people here have stories about how such
> conversations went for them, which arguments worked best, which
> didn't.

Thanks for your interest!

Currently, there is no such stories, AFAIK.  We discussed something
like that when releasing v1.2 but we had not been able to make it.

Well, if you are French speaker, a two mornings* workshop is in the
pipe.  The announce should come really soon.  The idea is sharing
feedback with other users from sysadmin to researcher.

*morning in metropolitan France meaning. :-)

> As a first step I would like to discuss with the sysadmins if they
> would consider activating user namespaces so users can play
> around. Aside from [1] and [2], are there any updated discussions
> about this topic, e.g. why to do it or why not?

You mean to run "bundles" on computers that do not have Guix installed, right?
Well, the answer "qui peut le plus peut le moins" is maybe not
satisfactoring... if you have the choice to do it, you should.  Even,
if you have the choice to install Guix (the package manager), you also
should.


Cheers,
simon

Re: Introducing Guix to HPC at my institution

[Sébastien Lerique]

Hi Simon,

Thanks for your answer!

On 15 Mar 2021 at 22:47, zimoun <zimon.toutoune@gmail.com> wrote:
> Well, if you are French speaker, a two mornings* workshop is in 
> the
> pipe.  The announce should come really soon.  The idea is 
> sharing
> feedback with other users from sysadmin to researcher.
>
> *morning in metropolitan France meaning. :-)

I am French, I'll be very happy to attend if I can! I live in 
Japan now though, so I guess I'll miss parts. Are you planning to 
record the workshop?

Another thought: given Ludo's position, I am imagining that Inria 
Bordeaux runs Guix on some of their infrastructure, is that the 
case? If so, can you share any details about how that came to 
happen?

>> As a first step I would like to discuss with the sysadmins if 
>> they
>> would consider activating user namespaces so users can play
>> around. Aside from [1] and [2], are there any updated 
>> discussions
>> about this topic, e.g. why to do it or why not?
>
> You mean to run "bundles" on computers that do not have Guix 
> installed, right?
> Well, the answer "qui peut le plus peut le moins" is maybe not
> satisfactoring... if you have the choice to do it, you should. 
> Even,
> if you have the choice to install Guix (the package manager), 
> you also
> should.

Yes I meant that. They run CentOS 8, so convincing them to run a 
build daemon will not be as simple as `apt install guix`, but 
we'll see how the conversation goes :)

I'll write back here in a few days once I've thought about how to 
present things to the sysadmins. In the meantime, any feedback on 
past experience is very welcome!


Sébastien

Re: Introducing Guix to HPC at my institution

[zimoun]

Hi Sábastien,

On Tue, 16 Mar 2021 at 10:54, Sébastien Lerique <sl@eauchat.org> wrote:

> On 15 Mar 2021 at 22:47, zimoun <zimon.toutoune@gmail.com> wrote:

> I am French, I'll be very happy to attend if I can! I live in
> Japan now though, so I guess I'll miss parts. Are you planning to
> record the workshop?

Cool!  If I remember correctly, there is some Japaneses’s users.  Do not
hesitate to roam on #guix (irc.freenode.net) or drop an email to
help-guix. :-)

We have not discussed the record of the workshop yet.  Keep you in
touch.

Well, I do not know if it is visible all around the world, here some
materials: 

<https://calcul.math.cnrs.fr/2021-01-anf-ust4hpc-2021.html#programme>

If I may, I would recommend the Konrad’s talk because it explains why
reproducibility matters for scientific reproducibility.  The next talk
by Ludo provides good connections with these principles and how to do in
practise.  I also recommend the 2 last talks by Francois Rué et Florent
Pruvost.  Other materials are worth to watch when speaking about
reproducibility, for sure, but they are not with a Guix angle.  Last, I
did not run the «TPs» so I cannot tell more.

Ludo’s did a nice talk at FOSDEM 2019 about distro:

<https://archive.fosdem.org/2019/schedule/event/gnu_guix_new_approach_to_software_distribution/>

well, he presents some arguments why and how Guix addresses the
challenge.  Moreover, here

<https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/talks>

you can find some raw materials used in various talks.  Especially one
at CERN in 2018 for instance:

<https://cds.cern.ch/record/2316926>



> Another thought: given Ludo's position, I am imagining that Inria
> Bordeaux runs Guix on some of their infrastructure, is that the
> case? If so, can you share any details about how that came to
> happen?

I cannot share how it runs at INRIA Bordeaux since I am far from my
beloved South West. :-) Well, some words how it runs at my place in a
small team of a Research Institute doing stuff in the biomedical field.

First, I am trained in apply maths and I studied wave propagation
(numerical methods, linear solver, etc.).  At the time, I am using
mainly Conda and some modulefiles prepared by the cluster’s team of the
different institutes I were.

Then, I traveled back to home and found a permanent position in this
Institute.  And I deal with Biologists with few skills on how to run
their computations.  I inherited a small cluster and couple of strong
desktop machines with an history.  Therefore, at the beginning, I was
doing the modulefiles by hand… then I said stop!  It does not scale
since I am alone and I felt as reinventing the wheel: repackaging
everything by hand.  Conda is still an option but Conda is doing wrong
at different levels.

I was following Guix, as the Emacs of distro. :-) And on Dec. 2018, Ludo
organized a day in Paris and I attended.  Then I installed Guix at work
on my main desktop and another remote strong desktop.  I teach Guix to
the new users and almost support only Guix.  But you know how people
have their habits. :-) One and half year later (2020 is off, since I was
recovering from a sport’s injury), I convinced people to switch to Guix
and I am in the process to reinstall the small cluster (for historical
reasons, LTS and never upgraded with too old kernel and now impossible
to upgrade “for free”, whatever!).

On strong desktops (256G of RAM, 64 cores) running a classic distro as
Ubuntu or Debian, installing Guix on the top is straightforward.  I do
as much as I can on the machines I am in charge.

Why Guix?  Beyond reproducibility, it is easy to have R packages via
“guix import”, for instance.  For the record, a Medical Doctor is
writing their own packages.  Well, people are less waiting after me and
they install the packages they need.  Guix works from laptop to cluster,
the surprises are really mitigated compared to the other tools I know.
Last, for users a bit afraid by the command line, I pack a Docker image
all included for them. Now a lot of journals ask code and data and I
would like to convince people in my Institute that
channels.scm+manifest.scm capture and Docker is just the container, easy
to move from one place to another; it has not happened yet.  That’s why
I investigated in things like:

<https://yhetil.org/guix/86bldahz42.fsf@gmail.com>


Last, Guix is a Scheme library.  Aside the parenthesis which is just not
an issue for a good editor, being Lisp allows to easily hack in.  Once
one is a bit familiar with a Lisp (Emacs Lisp, Racket, Common Lisp,
whatever, not necessary Guile) by running a tutorial, it is relatively
easy to read the code and understand which part does what.  For
instance, my first contribution after the usual packaging is fixing a
minor bug in the relevance scoring function.  I never understood Conda
enough to be able to fix any annoyance that I had.



> Yes I meant that. They run CentOS 8, so convincing them to run a
> build daemon will not be as simple as `apt install guix`, but
> we'll see how the conversation goes :)

Good luck!  Please report their feedback if they are not convinced.


> I'll write back here in a few days once I've thought about how to
> present things to the sysadmins. In the meantime, any feedback on
> past experience is very welcome!

I guess you already know the section «Clusters Deployment»:

<https://hpc.guix.info/about/>

Cheers,
simon

Re: Introducing Guix to HPC at my institution

[Ludovic Courtès]

Hi Sébastien,

Sébastien Lerique <sl@eauchat.org> skribis:

> Another thought: given Ludo's position, I am imagining that Inria
> Bordeaux runs Guix on some of their infrastructure, is that the 
> case? If so, can you share any details about how that came to happen?

I’ve been working with HPC people at Inria and back then I was working
on Guix in my spare time.  The HPC folks had been using “environment
modules” forever but were looking for a solution that would provide
automation, primarily, and optionally some kind of reproducibility.

They invested into Spack first (actually based on my recommendation; at
the time I wasn’t sure Guix could meet all their requirements) and grew
dissatisfied after a couple of years, in particular because
reproducibility became a more important factor for them.  It’s also
about the time when Ricardo and I published
<https://hal.inria.fr/hal-01161771/en> (Ricardo already had experience
with Guix in HPC and surely has an interesting story to share :-)).

We kept discussing these matters, together with sysadmins of the local
cluster, who were also looking for a solution they could offer users in
addition to modules.  Eventually Guix was installed on that cluster and
local HPC teams got into it at that point.

That’s my story!  :-)

It’s hopefully not the end of the story though.  There are other
clusters listed at <https://hpc.guix.info/about/> that provide Guix.
We’re discussing with others in France, but these things take time…

> Yes I meant that. They run CentOS 8, so convincing them to run a build
> daemon will not be as simple as `apt install guix`, but we'll see how
> the conversation goes :)

Tools like modules, Conda, Spack, and “containers” (Docker, Singularity)
are quite entrenched now in HPC.  All these tools are relatively new so
proposing yet another tool can be a hard sell.

Also, cluster sysadmins tend to be conservative and don’t like the idea
of having a “daemon running as root”.

But my impression is that there’s increasing awareness of the
limitations of these tools.  It used to be that people would disagree
when we said in talks that Docker/Singularity images are not
reproducible; now this is more widely understood.  Likewise for the
deployment issues around Jupyter and workflow languages.

And there’s not a zillion tools out there to address these issues.  :-)

Good luck with your endeavors!

Ludo’.

Re: Introducing Guix to HPC at my institution

[Sébastien Lerique]

Hi Ludo, Simon,

Thanks for these very interesting stories! That's a lot of 
resources too, I will be watching some of the talks you linked to 
over the coming days. I'm currently contacting people who might 
already feel a strong need for what Guix can offer, to see what 
would be the best way to introduce the topic to the sysadmins. Who 
knows, maybe my institution can also support the development of 
Guix in some way :)

Will be back once I have some progress or more questions.

Best,
Sébastien

On 16 Mar 2021 at 18:05, Ludovic Courtès 
<ludovic.courtes@inria.fr> wrote:

> Hi Sébastien,
>
> Sébastien Lerique <sl@eauchat.org> skribis:
>
>> Another thought: given Ludo's position, I am imagining that 
>> Inria
>> Bordeaux runs Guix on some of their infrastructure, is that the
>> case? If so, can you share any details about how that came to 
>> happen?
>
> I’ve been working with HPC people at Inria and back then I was 
> working
> on Guix in my spare time.  The HPC folks had been using 
> “environment
> modules” forever but were looking for a solution that would 
> provide
> automation, primarily, and optionally some kind of 
> reproducibility.
>
> They invested into Spack first (actually based on my 
> recommendation; at
> the time I wasn’t sure Guix could meet all their requirements) 
> and grew
> dissatisfied after a couple of years, in particular because
> reproducibility became a more important factor for them.  It’s 
> also
> about the time when Ricardo and I published
> <https://hal.inria.fr/hal-01161771/en> (Ricardo already had 
> experience
> with Guix in HPC and surely has an interesting story to share 
> :-)).
>
> We kept discussing these matters, together with sysadmins of the 
> local
> cluster, who were also looking for a solution they could offer 
> users in
> addition to modules.  Eventually Guix was installed on that 
> cluster and
> local HPC teams got into it at that point.
>
> That’s my story!  :-)
>
> It’s hopefully not the end of the story though.  There are other
> clusters listed at <https://hpc.guix.info/about/> that provide 
> Guix.
> We’re discussing with others in France, but these things take 
> time…
>
>> Yes I meant that. They run CentOS 8, so convincing them to run 
>> a build
>> daemon will not be as simple as `apt install guix`, but we'll 
>> see how
>> the conversation goes :)
>
> Tools like modules, Conda, Spack, and “containers” (Docker, 
> Singularity)
> are quite entrenched now in HPC.  All these tools are relatively 
> new so
> proposing yet another tool can be a hard sell.
>
> Also, cluster sysadmins tend to be conservative and don’t like 
> the idea
> of having a “daemon running as root”.
>
> But my impression is that there’s increasing awareness of the
> limitations of these tools.  It used to be that people would 
> disagree
> when we said in talks that Docker/Singularity images are not
> reproducible; now this is more widely understood.  Likewise for 
> the
> deployment issues around Jupyter and workflow languages.
>
> And there’s not a zillion tools out there to address these 
> issues.  :-)
>
> Good luck with your endeavors!
>
> Ludo’.

Re: Introducing Guix to HPC at my institution

[Sébastien Lerique]

Hi Ludo, Simon, all,

> Will be back once I have some progress or more questions.

It turns out the HPC cluster I have access to has user namespaces 
activated \o/, so I'm looking into getting things running as an 
unpriviliged user to show other people how useful Guix can be 
(before approaching higher levels in the administration).

I have been through the following notes:

https://hpc.guix.info/blog/2017/09/reproducibility-and-root-privileges/
https://hpc.guix.info/blog/2017/10/using-guix-without-being-root/
http://issues.guix.gnu.org/34494

and am now following Guix's binary installation inside a user 
namespace. After decompressing the binary distribution of guix 
inside `~/local-guix`, my naïve next step was `unshare -mrf chroot 
~/local-guix 
gnu/store/mmhimfwmmidf09jw1plw3aw1g1zn2nkh-bash-static-5.0.16/bin/bash`.

But my knowledge of linux namespaces is hindering my next steps 
:). A few questions:

- after setting $GUIX_PROFILE and sourcing 
  `/root/.config/guix/current`, running `guix` warns with:

  GC Warning: pthread_getattr_np or pthread_attr_getstack failed 
  for main thread
  GC Warning: Couldn't read /proc/stat

  The first warning I don't know what to do with. About the 
  second: should I be binding `/proc` somehow?

- is it possible to create build users inside the user-namespaced 
  chroot?

- last but not least, how would I go about sharing this setup with 
  other users on the cluster? Ideally I would like to have a 
  non-priviliged build daemon that other users can call on. (Is 
  there such a thing as kernel group namespaces?)

Is this the right way to go for running guix without being root, 
or is there a better way?

Thanks for any guidance you might provide!
Best,
Sébastien

Re: Introducing Guix to HPC at my institution

[Ludovic Courtès]

Hi Sébastien,

Sébastien Lerique <sl@eauchat.org> skribis:

> It turns out the HPC cluster I have access to has user namespaces
> activated \o/, so I'm looking into getting things running as an 
> unpriviliged user to show other people how useful Guix can be (before
> approaching higher levels in the administration).

Good!

> and am now following Guix's binary installation inside a user
> namespace. After decompressing the binary distribution of guix 
> inside `~/local-guix`, my naïve next step was `unshare -mrf chroot
> ~/local-guix 
> gnu/store/mmhimfwmmidf09jw1plw3aw1g1zn2nkh-bash-static-5.0.16/bin/bash`.

Instead of installing the “regular” binary tarball inside a namespace,
it might be easier to create a tarball like so:

  guix pack -RR -S /bin=bin -S /etc=etc guix bash

… and to unpack the resulting tarball.

From there, you can run ./bin/sh to get a shell that “sees” /gnu/store.
You can then run:

  . ./etc/profile

And then, you should be able to run the daemon, like so:

  export GUIX_STATE_DIRECTORY=$HOME/.local/var/guix
  guix-daemon --disable-chroot &

(Adapted from
<https://lists.gnu.org/archive/html/guix-devel/2018-05/msg00139.html>.)

Does that work for you?

> But my knowledge of linux namespaces is hindering my next steps :). A
> few questions:
>
> - after setting $GUIX_PROFILE and sourcing
>   `/root/.config/guix/current`, running `guix` warns with:
>
>  GC Warning: pthread_getattr_np or pthread_attr_getstack failed   for
> main thread
>  GC Warning: Couldn't read /proc/stat
>
>  The first warning I don't know what to do with. About the   second:
> should I be binding `/proc` somehow?

Yes, you should expose /proc.  The wrappers created by ‘guix pack -RR’
in the example above bind-mount everything + /gnu/store, such that you
can’t tell the difference.

> - is it possible to create build users inside the user-namespaced
>   chroot?

No: you still have a single UID at hand, so there’s no way to allocate
new ones.

> - last but not least, how would I go about sharing this setup with
>   other users on the cluster? Ideally I would like to have a 
>  non-priviliged build daemon that other users can call on. (Is   there
> such a thing as kernel group namespaces?)

It’s not really sharable.  To share it, you would need some sort of a
shared trusted “proxy”; that’s precisely what guix-daemon is in normal
multi-user setups.

HTH!

Ludo’.

Re: Introducing Guix to HPC at my institution

[Sébastien Lerique]

Hi Ludo,

Thanks for the help!

On 29 Mar 2021 at 21:03, Ludovic Courtès 
<ludovic.courtes@inria.fr> wrote:

> Instead of installing the “regular” binary tarball inside a 
> namespace,
> it might be easier to create a tarball like so:
>
>   guix pack -RR -S /bin=bin -S /etc=etc guix bash
>
> … and to unpack the resulting tarball.
>
> From there, you can run ./bin/sh to get a shell that “sees” 
> /gnu/store.
> You can then run:
>
>   . ./etc/profile
>
> And then, you should be able to run the daemon, like so:
>
>   export GUIX_STATE_DIRECTORY=$HOME/.local/var/guix
>   guix-daemon --disable-chroot &
>
> (Adapted from
> <https://lists.gnu.org/archive/html/guix-devel/2018-05/msg00139.html>.)
>
> Does that work for you?

Thanks! I `guix pack`'ed with a single -R since I shouldn't need 
PRoot, and had to add

  export GUIX_LOG_DIRECTORY=$HOME/.local/var/log

to the environment setup in order to start guix-daemon. Then `guix 
install hello` warned at the beginning with

  user with UID 7352 not found

and later failed with

  Backtrace:
  In ice-9/boot-9.scm:
    1736:10  6 (with-exception-handler _ _ #:unwind? _ # _)
  In unknown file:
             5 (apply-smob/0 #<thunk 7f5a32f85520>)
  In ice-9/boot-9.scm:
      718:2  4 (call-with-prompt _ _ #<procedure 
      default-prompt-handle…>)
  In ice-9/eval.scm:
      619:8  3 (_ #(#(#<directory (guile-user) 7f5a32f88c80>)))
  In guix/ui.scm:
    2164:12  2 (run-guix-command _ . _)
  In guix/scripts/offload.scm:
     782:21  1 (guix-offload "x86_64-linux" "0" "1" "0")
  In unknown file:
             0 (getpw 7352)

  ERROR: In procedure getpw:
  In procedure getpw: entry not found
  killing process 9106
  guix install: error: unexpected EOF reading a line

I'm guessing some part on the non-namespaced uids are leaking into 
the user namespace?


Another problem appeared for substitutes (as 'guix install hello' 
was rebuilding the world):

  $ guix archive --authorize < \
    /gnu/store/xb4szjambyi52bpnkjv080g2mlfqqpp0-profile/share/guix/ci.guix.gnu.org.pub
  guix archive: error: mkdir: Permission denied


I quickly went through `guix/scripts/pack.scm` and 
`gnu/packages/aux-files/run-in-namespace.c` to try and understand 
what `guix pack -R` does, but it looks like I need to learn more 
about namespaces before I can digest that. My high-level 
understanding from poking around is that the binaries from `guix 
pack -R` run inside a namespace where /gnu is bind-mounted (along 
with /proc, /dev, /sys, ...), but the rest of the host filesystem 
is still available, and other host processes can also be seen. 
Then, anything I run from inside the `guix pack`'ed bash inherits 
that namespace, which is what makes guix-daemon work.

If this is incorrect, could you maybe give a high-level overview 
of what it does? The documentation is a bit scarce on the topic 
(or I didn't find it).

I'm also wondering why `guix-daemon` must be invoked with 
`--disable-chroot`. Doesn't that make the reproducibility 
guarantees more brittle (according to the docs)?

>> - is it possible to create build users inside the 
>> user-namespaced
>>   chroot?
>
> No: you still have a single UID at hand, so there’s no way to 
> allocate
> new ones.
>
>> - last but not least, how would I go about sharing this setup 
>> with
>>   other users on the cluster? Ideally I would like to have a
>>  non-priviliged build daemon that other users can call on. (Is 
>>  there
>> such a thing as kernel group namespaces?)
>
> It’s not really sharable.  To share it, you would need some sort 
> of a
> shared trusted “proxy”; that’s precisely what guix-daemon is in 
> normal
> multi-user setups.

Ok. My use cases here are:
- trying to provide a setup that others can more-or-less easily 
  try out to get a feel for Guix on the HPC cluster (if possible 
  without duplicating /gnu/store), thus creating interest and user 
  support for later presenting the case to sysadmins / university 
  administration;
- getting to know the exact requirements that I would have to 
  convince sysadmins to agree to, with rationales/explanations for 
  the hardest-to-obtain. Getting them to run guix-daemon as root 
  is probably a lost case, so I'd like to explore everything else 
  that is possible.

So let's transform my two questions above into the two following:

1. what would be the list of requirements that explain why 
guix-daemon must run as root? What does each of those requirements 
accomplish?
2. knowing that my environment has user namespaces (and knowing 
that that may not be enough), would it be possible to set up 
guix-daemon as non-root with the assistance of a sysadmin (so say 
with root access during setup), with build users and all, and have 
it provide identical guarantees to a guix-daemon running as root? 
If not, why not, and could we work around that?


Thanks again for all the help!
Sébastien

Re: Introducing Guix to HPC at my institution

[Ludovic Courtès]

Hi Sébastien,

Sébastien Lerique <sl@eauchat.org> skribis:

> On 29 Mar 2021 at 21:03, Ludovic Courtès <ludovic.courtes@inria.fr>
> wrote:
>
>> Instead of installing the “regular” binary tarball inside a
>> namespace,
>> it might be easier to create a tarball like so:
>>
>>   guix pack -RR -S /bin=bin -S /etc=etc guix bash
>>
>> … and to unpack the resulting tarball.

[...]

> Thanks! I `guix pack`'ed with a single -R since I shouldn't need
> PRoot, and had to add
>
>  export GUIX_LOG_DIRECTORY=$HOME/.local/var/log

OK.

> to the environment setup in order to start guix-daemon. Then `guix
> install hello` warned at the beginning with
>
>  user with UID 7352 not found
>
> and later failed with
>
>  Backtrace:
>  In ice-9/boot-9.scm:
>    1736:10  6 (with-exception-handler _ _ #:unwind? _ # _)
>  In unknown file:
>             5 (apply-smob/0 #<thunk 7f5a32f85520>)
>  In ice-9/boot-9.scm:
>      718:2  4 (call-with-prompt _ _ #<procedure
>      default-prompt-handle…>)
>  In ice-9/eval.scm:
>      619:8  3 (_ #(#(#<directory (guile-user) 7f5a32f88c80>)))
>  In guix/ui.scm:
>    2164:12  2 (run-guix-command _ . _)
>  In guix/scripts/offload.scm:
>     782:21  1 (guix-offload "x86_64-linux" "0" "1" "0")
>  In unknown file:
>             0 (getpw 7352)
>
>  ERROR: In procedure getpw:
>  In procedure getpw: entry not found

Hmm could it be that nscd is not running, and that /etc/nsswitch.conf
specifies a “non-standard” NSS plugin, such as ‘sssd’?

  https://guix.gnu.org/manual/en/html_node/Application-Setup.html#Name-Service-Switch-1

Does ‘guix install hello --no-offload’ work around the issue?

> Another problem appeared for substitutes (as 'guix install hello' was
> rebuilding the world):
>
>  $ guix archive --authorize < \
>    /gnu/store/xb4szjambyi52bpnkjv080g2mlfqqpp0-profile/share/guix/ci.guix.gnu.org.pub
>  guix archive: error: mkdir: Permission denied

I guess it’s trying to write to /etc/guix.  You probably need to set
GUIX_STATE_DIRECTORY=$HOME/.local/var/guix here.

> I quickly went through `guix/scripts/pack.scm` and
> `gnu/packages/aux-files/run-in-namespace.c` to try and understand 
> what `guix pack -R` does, but it looks like I need to learn more about
> namespaces before I can digest that. My high-level understanding from
> poking around is that the binaries from `guix pack -R` run inside a
> namespace where /gnu is bind-mounted (along with /proc, /dev, /sys,
> ...), but the rest of the host filesystem is still available, and
> other host processes can also be seen. Then, anything I run from
> inside the `guix pack`'ed bash inherits that namespace, which is what
> makes guix-daemon work.

Correct!

> If this is incorrect, could you maybe give a high-level overview of
> what it does? The documentation is a bit scarce on the topic (or I
> didn't find it).

The design and implementation are documented in blog posts:

  https://guix.gnu.org/en/blog/2018/tarballs-the-ultimate-container-image-format/
  https://hpc.guix.info/blog/2020/05/faster-relocatable-packs-with-fakechroot/

> I'm also wondering why `guix-daemon` must be invoked with
> `--disable-chroot`. Doesn't that make the reproducibility 
> guarantees more brittle (according to the docs)?

True, but I think (?) that’s currently unavoidable in this setup…
though I forget why.  You might want to try without ‘--disable-chroot’.

> Ok. My use cases here are:
> - trying to provide a setup that others can more-or-less easily   try
>  out to get a feel for Guix on the HPC cluster (if possible   without
> duplicating /gnu/store), thus creating interest and user   support for
> later presenting the case to sysadmins / university   administration;

Safe sharing among users is not possible without a daemon coordinating
accesses to the store and acting as a trusted proxy.

> - getting to know the exact requirements that I would have to
>   convince sysadmins to agree to, with rationales/explanations for 
>  the hardest-to-obtain. Getting them to run guix-daemon as root   is
> probably a lost case, so I'd like to explore everything else   that is
> possible.

Yeah.

> So let's transform my two questions above into the two following:
>
> 1. what would be the list of requirements that explain why guix-daemon
> must run as root? What does each of those requirements accomplish?

I think this article remains relevant:

  https://hpc.guix.info/blog/2017/09/reproducibility-and-root-privileges/

> 2. knowing that my environment has user namespaces (and knowing that
> that may not be enough), would it be possible to set up guix-daemon as
> non-root with the assistance of a sysadmin (so say with root access
> during setup), with build users and all, and have it provide identical
> guarantees to a guix-daemon running as root? If not, why not, and
> could we work around that?

Good question.  It should be possible to make the daemon run as
non-root; that’s what the trick with the ‘guix pack -R’ wrapper should
achieve, but it could also be a built-in capability.

Food for thought!

Thanks,
Ludo’.

Re: Introducing Guix to HPC at my institution

[Sébastien Lerique]

Hi Ludo, zimoun, all,

Thanks again for your answers, for the links to documentation, and 
for the support on IRC!

> Hmm could it be that nscd is not running, and that 
> /etc/nsswitch.conf
> specifies a “non-standard” NSS plugin, such as ‘sssd’?
>
>   https://guix.gnu.org/manual/en/html_node/Application-Setup.html#Name-Service-Switch-1
>
> Does ‘guix install hello --no-offload’ work around the issue?

Yes, that solves the issue!

>>  $ guix archive --authorize < \
>>    /gnu/store/xb4szjambyi52bpnkjv080g2mlfqqpp0-profile/share/guix/ci.guix.gnu.org.pub
>>  guix archive: error: mkdir: Permission denied
>
> I guess it’s trying to write to /etc/guix.  You probably need to 
> set
> GUIX_STATE_DIRECTORY=$HOME/.local/var/guix here.

Yes, that would be GUIX_CONFIGURATION_DIRECTORY. As discussed on 
IRC though, /etc in the tarball is a symlink to the profile, so 
neither `guix archive` nor I can create /etc/guix. So I resorted 
to creating a `/etc-guix` folder which I use for 
GUIX_CONFIGURATION_DIRECTORY.

>> 2. knowing that my environment has user namespaces (and knowing 
>> that
>> that may not be enough), would it be possible to set up 
>> guix-daemon as
>> non-root with the assistance of a sysadmin (so say with root 
>> access
>> during setup), with build users and all, and have it provide 
>> identical
>> guarantees to a guix-daemon running as root? If not, why not, 
>> and
>> could we work around that?
>
> Good question.  It should be possible to make the daemon run as
> non-root; that’s what the trick with the ‘guix pack -R’ wrapper 
> should
> achieve, but it could also be a built-in capability.

Yes, I'd be very happy to try and build that once things work!

So, to summarize the current status, here is what I do:

On my Guix System (`deigo` is my ssh config for the cluster):

  $ scp (guix pack -R -S /bin=bin -S /etc=etc guix bash) 
  deigo:guix-pack.tar.gz

Then on deigo:

  $ mkdir -p /a-working-directory/guix
  $ cd /a-working-directory/guix
  $ tar xzf ~/guix-pack.tar.gz
  $ mkdir etc-guix
  $ bin/bash
  $ . etc/profile
  $ export GUIX_STATE_DIRECTORY=$(pwd)/var/guix
  $ export GUIX_LOG_DIRECTORY=$(pwd)/var/log
  $ export GUIX_CONFIGURATION_DIRECTORY=$(pwd)/etc-guix

Then the fun starts. There are 4 different cases and problems, 
depending on whether I use substitutes or not, and on whether 
`/a-working-directory` is an NFS share or a local mount.


Case 1: with substitutes, on a local (non-NFS) folder
-----------------------------------------------------

  $ guix archive --authorize < 
  bin/../share/guix/ci.guix.gnu.org.pub
  $ guix-daemon --disable-chroot &
  $ guix install hello
  [... dowloading ...]
  unexpected substituter message 'defining `GUIX_LOCPATH', along 
  these lines:

       guix install glibc-utf8-locales
       export GUIX_LOCPATH="$HOME/.guix-profile/lib/locale"

  See the "Application Setup" section in the manual, for more 
  info.
  '

I think this is https://issues.guix.gnu.org/45166.

Using instead `guix pack -R -S /bin=bin -S /etc=etc -S /lib=lib 
guix bash glibc-utf8-locales`, and:

  $ export GUIX_LOCPATH=$(pwd)/lib/locale
  $ ls $GUIX_LOCPATH/
  2.31

seems to start working, but then `guix install hello` fails with 
the same error, and:

  $ ls $GUIX_LOCPATH/
  ls: cannot access '/tmp/sebastien-lerique/guix/lib/locale/': No 
  such file or directory

Is it possible that glibc-utf8-locales gets garbage-collected?


Case 2: from source (no substitutes), on a local (non-NFS) folder
-----------------------------------------------------------------

No need for the locale fix here (another bug appears before that):

  $ guix-daemon --disable-chroot &
  # --no-offload works around the missing nscd problem; up to now 
  this
  # doesn't seem necessary when using substitutes
  $ guix install hello --no-offload
  [... builds for a while ...]
  build of 
  /gnu/store/pkn1w1q3xkn273kpmggc4dnq6n6hr9jy-bzip2-mesboot-1.0.8.drv 
  failed

The build log for bzip2-mesboot ends with:

  tcc -ar cq libbz2.a blocksort.o huffman.o crctable.o randtable.o 
  compress.o decompress.o bzlib.o
  ranlib libbz2.a
  /gnu/store/prkqai3zwh3shlqpll6xyncmmqpj49dd-gash-boot-0.2.0/bin/sh: 
  ranlib: Command not found.
  make: *** [libbz2.a] Error 127
  command "make" "CC=tcc -I ." "AR=tcc -ar" "bzip2" 
  "PREFIX=/gnu/store/s94hyrv1vgllxir5niiyzfc9g80l5kcd-bzip2-mesboot-1.0.8" 
  failed with status 2

Is this new and should be reported as a bug? Should 
binutils-mesboot0 be part of bzip2-mesboot's inputs? (I haven't 
learned about the boostrapping mechanism yet.)


Case 3: with substitutes, on an NFS share
-----------------------------------------

Again no need for the locale fix (another bug appears before 
that):

  $ guix archive --authorize < 
  bin/../share/guix/ci.guix.gnu.org.pub
  $ guix-daemon --disable-chroot &
  $ guix install hello
  [... downloading ...]
  guix install: error: cannot unlink 
  `/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib': 
  Directory not empty

so checking:

  $ ls -lA 
  /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib
  total 3840
  -r-xr-xr-x 1 sebastien-lerique froeseuni   64832 Jan  1  1970 
  .nfs000000000172caa5000054bd
  [... more .nfs files ...]

  $ lsof -f -- 
  /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib/.nfs000000000172caa5000054bd` 
  reports:
  [... some warnings ...]
  COMMAND     PID              USER  FD   TYPE DEVICE SIZE/OFF 
  NODE NAME
  bash      25616 sebastien-lerique mem    REG   0,52    64832 
  24300197 
  /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib/.nfs000000000172caa5000054bd
  guix-daem 25934 sebastien-lerique mem    REG   0,52    64832 
  24300197 
  /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib/.nfs000000000172caa5000054bd

Not sure what's going wrong here. Maybe it's more 
garbage-collecting (as in case 1), failing because of an nfs 
nuance or bug?


Case 4: from source (no substitutes), on an NFS share
-----------------------------------------------------

Fails in the exact same way as Case 2.


I have no idea how to make progress in any of these cases (except 
the bzip2-mesboot one maybe), so any input on these different 
problems is very welcome :)

Thanks and happy hacking!
Sébastien

Re: Introducing Guix to HPC at my institution

[Ludovic Courtès]

Howdy!

Sébastien Lerique <sl@eauchat.org> skribis:

> Case 1: with substitutes, on a local (non-NFS) folder
> -----------------------------------------------------
>
>  $ guix archive --authorize <   bin/../share/guix/ci.guix.gnu.org.pub
>  $ guix-daemon --disable-chroot &
>  $ guix install hello
>  [... dowloading ...]
>  unexpected substituter message 'defining `GUIX_LOCPATH', along
>  these lines:
>
>       guix install glibc-utf8-locales
>       export GUIX_LOCPATH="$HOME/.guix-profile/lib/locale"
>
>  See the "Application Setup" section in the manual, for more   info.
>  '
>
> I think this is https://issues.guix.gnu.org/45166.
>
> Using instead `guix pack -R -S /bin=bin -S /etc=etc -S /lib=lib guix
> bash glibc-utf8-locales`, and:
>
>  $ export GUIX_LOCPATH=$(pwd)/lib/locale
>  $ ls $GUIX_LOCPATH/
>  2.31
>
> seems to start working, but then `guix install hello` fails with the
> same error, and:
>
>  $ ls $GUIX_LOCPATH/
>  ls: cannot access '/tmp/sebastien-lerique/guix/lib/locale/': No
>  such file or directory
>
> Is it possible that glibc-utf8-locales gets garbage-collected?

Can you run guix-daemon like so?

  LC_ALL=en_US.utf8 guix-daemon --disable-chroot &

The “unexpected substitute message” thing is a bug: it turns out that
stderr of ‘guix substitute’ is consumed directly by the daemon at this
point, which it shouldn’t (similar to the issue fixed by
ee3226e9d54891c7e696912245e4904435be191c).

> Case 2: from source (no substitutes), on a local (non-NFS) folder
> -----------------------------------------------------------------
>
> No need for the locale fix here (another bug appears before that):
>
>  $ guix-daemon --disable-chroot &
>  # --no-offload works around the missing nscd problem; up to now
>      this
>  # doesn't seem necessary when using substitutes
>  $ guix install hello --no-offload
>  [... builds for a while ...]
>  build of
>  /gnu/store/pkn1w1q3xkn273kpmggc4dnq6n6hr9jy-bzip2-mesboot-1.0.8.drv 
>  failed
>
> The build log for bzip2-mesboot ends with:
>
>  tcc -ar cq libbz2.a blocksort.o huffman.o crctable.o randtable.o
>  compress.o decompress.o bzlib.o
>  ranlib libbz2.a
>  /gnu/store/prkqai3zwh3shlqpll6xyncmmqpj49dd-gash-boot-0.2.0/bin/sh:
>  ranlib: Command not found.
>  make: *** [libbz2.a] Error 127
>  command "make" "CC=tcc -I ." "AR=tcc -ar" "bzip2"
>  "PREFIX=/gnu/store/s94hyrv1vgllxir5niiyzfc9g80l5kcd-bzip2-mesboot-1.0.8" 
>  failed with status 2
>
> Is this new and should be reported as a bug? Should binutils-mesboot0
> be part of bzip2-mesboot's inputs? (I haven't learned about the
> boostrapping mechanism yet.)

The problem is that ‘--disable-chroot’ is a bit of the wild west: build
processes can access the whole file system and what you do as a user can
interfere with them.

It could be that the bzip2 build failure above is just that: the build
process picks something from /usr/lib or /usr/bin, and that breaks
everything.

I think ‘--disable-chroot’ is OK if you’re going to use substitutes for
almost everything.  Otherwise, it’s not good.  Your use case calls for
built-in support; that way, the daemon could take still advantage of
user namespaces to set up a chroot and all.

> Case 3: with substitutes, on an NFS share
> -----------------------------------------
>
> Again no need for the locale fix (another bug appears before that):
>
>  $ guix archive --authorize <   bin/../share/guix/ci.guix.gnu.org.pub
>  $ guix-daemon --disable-chroot &
>  $ guix install hello
>  [... downloading ...]
>  guix install: error: cannot unlink
>  `/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib': 
>  Directory not empty
>
> so checking:
>
>  $ ls -lA   /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib
>  total 3840
>  -r-xr-xr-x 1 sebastien-lerique froeseuni   64832 Jan  1  1970
>   .nfs000000000172caa5000054bd
>  [... more .nfs files ...]

Ah yes, that’s a known issue with NFS.  I’m not aware of any workaround.

Thanks,
Ludo’.

Re: Introducing Guix to HPC at my institution

[Sébastien Lerique]

Hello!

On 01 Apr 2021 at 17:35, Ludovic Courtès 
<ludovic.courtes@inria.fr> wrote:

>> Case 1: with substitutes, on a local (non-NFS) folder
>> -----------------------------------------------------
>> [snip]
>
> Can you run guix-daemon like so?
>
>   LC_ALL=en_US.utf8 guix-daemon --disable-chroot &

Then:

  $ guix install hello --no-offload
  [... dowloading ...]
  13.3 MB will be downloaded
  substitution of 
  /gnu/store/395pvii4bcjqxvdv7h0drq10lxi01sv1-glibc-utf8-locales-2.31 
  failed
  guix install: error: some substitutes for the outputs of 
  derivation 
  `/gnu/store/b2jkmg71m0dpf3i6hvskb32ra48lls28-glibc-utf8-locales-2.31.drv' 
  failed (usually happens due to networking issues); try 
  `--fallback' to build derivation from source

Restarting from scratch and using `--fallback` then leads to the 
bzip2-mesboot failure of Case 2.

If, instead of resarting from scratch, I just run again `guix 
install hello --no-offload`, I then get:

  guix install: error: got unexpected path 
  `/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16/bin/bash: 
  warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)' 
  from substituter

And indeed once again lib/locale is a dead symlink, as the 
glibc-utf8-locales packaged in my tar.gz has been removed from the 
store. Why is that? I had a look through the same commands with 
--debug=4 or 5, but to no insight (aside from seeing locale 
derivations being deleted at some stage). I posted the end of the 
log produced by `guix install hello --no-offload --debug=4`, 
starting with the failed substitution, here: 
https://paste.debian.net/1191991/ .

> The “unexpected substitute message” thing is a bug: it turns out 
> that
> stderr of ‘guix substitute’ is consumed directly by the daemon 
> at this
> point, which it shouldn’t (similar to the issue fixed by
> ee3226e9d54891c7e696912245e4904435be191c).

Ok, I'll file a bug for that then :)

>> Case 2: from source (no substitutes), on a local (non-NFS) 
>> folder
>> -----------------------------------------------------------------
>> [snip]
>
> The problem is that ‘--disable-chroot’ is a bit of the wild 
> west: build
> processes can access the whole file system and what you do as a 
> user can
> interfere with them.
>
> It could be that the bzip2 build failure above is just that: the 
> build
> process picks something from /usr/lib or /usr/bin, and that 
> breaks
> everything.
>
> I think ‘--disable-chroot’ is OK if you’re going to use 
> substitutes for
> almost everything.  Otherwise, it’s not good.  Your use case 
> calls for
> built-in support; that way, the daemon could take still 
> advantage of
> user namespaces to set up a chroot and all.

I see. By the way, starting guix-daemon without `--disable-chroot` 
(and without substitutes) leads to:

  guix build: error: cannot change ownership of 
  ‘/gnu/store/0dn61y4n8ig333b23hmc80hvlcy8gdli-guile-bootstrap-2.0.drv.chroot’: 
  Invalid argument

so --disable-chroot is indeed still necessary! I am interested in 
working on this built-in support once I get the substitutes case 
working (although I will probably come ask for guidance for that) 
:)

Thanks!
Sébastien

Re: Introducing Guix to HPC at my institution

[Ludovic Courtès]

Hi Sébastien,

Sébastien Lerique <sl@eauchat.org> skribis:

> If, instead of resarting from scratch, I just run again `guix install
> hello --no-offload`, I then get:
>
>  guix install: error: got unexpected path
>  `/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16/bin/bash: 
>  warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)'   from
> substituter

I (re-)fixed this particular problem here:

  https://issues.guix.gnu.org/46362

So if you create a pack containing the ‘guix’ package from current
master, this problem should no longer be there.

Ludo’.

Re: Introducing Guix to HPC at my institution

[Sébastien Lerique]

Hi Ludo,

On 11 Apr 2021 at 05:43, Ludovic Courtès <ludo@gnu.org> wrote:

>> If, instead of resarting from scratch, I just run again `guix 
>> install
>> hello --no-offload`, I then get:
>>
>>  guix install: error: got unexpected path
>>  `/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16/bin/bash:
>>  warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)' 
>>  from
>> substituter
>
> I (re-)fixed this particular problem here:
>
>   https://issues.guix.gnu.org/46362
>
> So if you create a pack containing the ‘guix’ package from 
> current
> master, this problem should no longer be there.

Success! With substitutes:

  $ guix install hello --no-offload
  [... installs ...]
  $ GUIX_PROFILE="/home/s/sebastien-lerique/.guix-profile"
  $ . "$GUIX_PROFILE/etc/profile"
  $ hello
  Hello, world!

Then my next steps are to try out a few more things, see if some 
builds work (based on existing substitutes, not from scratch), 
gather some interest here now that I can demo, and then come back 
to explore builtin support for a non-root shared build daemon.

Thank you!!
Sébastien

Re: Introducing Guix to HPC at my institution

[Ludovic Courtès]

Hi,

Sébastien Lerique <sl@eauchat.org> skribis:

> Success! With substitutes:
>
>  $ guix install hello --no-offload
>  [... installs ...]
>  $ GUIX_PROFILE="/home/s/sebastien-lerique/.guix-profile"
>  $ . "$GUIX_PROFILE/etc/profile"
>  $ hello
>  Hello, world!

Yay!  (Of course, “echo hello world” would have been simpler but not as
as fun.  :-))

> Then my next steps are to try out a few more things, see if some
> builds work (based on existing substitutes, not from scratch), 
> gather some interest here now that I can demo, and then come back to
> explore builtin support for a non-root shared build daemon.

Awesome, let us know how it goes!

Ludo’.