[Non-root Guix]: Unable to build guix

[Non-root Guix]: Unable to build guix

[rohit yadav]

[-- Attachment #1: Type: text/plain, Size: 4798 bytes --]

Hi,

First of all congratulations on releasing 1.0.0. I have been waiting for
this for a long time.

I am trying to prepare guix for installing at non-root location. I have
followed the documentation available here:
https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#error-in-procedure-scm-error-no-code-for-module-gcrypt-hash

For most part everything went find. However, when I want to build packages
for the non-root store, I find stuck at module crypt hash stage. I looked
at the old thread and tried multiple suggestions:
- Building in the pure environment
- exporting the guile paths etc
But for some reason the guix build process still fails to read the crypt
module. Here is the error message:

*#+begin_log*
./pre-inst-env guix package -i guix


substitute: Backtrace:


substitute: In ice-9/threads.scm:
substitute:     390:8 19 (_ _)
substitute: In ice-9/boot-9.scm:
substitute:   2994:20 18 (_)
substitute:    2312:4 17 (save-module-excursion _)
substitute:   3014:26 16 (_)
substitute: In unknown file:
substitute:           15 (primitive-load-path "guix/store" #<procedure
1bac900 a?>)
substitute: In guix/store.scm:
substitute:      20:0 14 (_)
substitute: In ice-9/boot-9.scm:
substitute:    2874:4 13 (define-module* _ #:filename _ #:pure _ #:version
_ # _ ?)
substitute:   2887:24 12 (_)
substitute:    222:29 11 (map1 (((guix utils)) ((guix config)) ((guix #))
((?)) ?))
substitute:    222:29 10 (map1 (((guix config)) ((guix deprecation)) ((guix
?)) ?))
substitute:    222:29  9 (map1 (((guix deprecation)) ((guix memoization))
((?)) ?))
substitute:    222:29  8 (map1 (((guix memoization)) ((guix serialization))
(#) ?))
substitute:    222:29  7 (map1 (((guix serialization)) ((guix monads)) ((#
#)) ?))
substitute:    222:29  6 (map1 (((guix monads)) ((guix records)) ((guix #))
(#) ?))
substitute:    222:29  5 (map1 (((guix records)) ((guix base16)) ((guix #))
(#) ?))
substitute:    222:29  4 (map1 (((guix base16)) ((guix base32)) ((gcrypt
#)) # ?))
substitute:    222:29  3 (map1 (((guix base32)) ((gcrypt hash)) ((guix #))
(#) ?))
substitute:    222:17  2 (map1 (((gcrypt hash)) ((guix profiling)) ((rnrs
#)) # ?))
substitute:    2803:6  1 (resolve-interface _ #:select _ #:hide _ #:prefix
_ # _ ?)
substitute: In unknown file:
substitute:            0 (scm-error misc-error #f "~A ~S" ("no code for
modu?" ?) ?)
substitute:
substitute: ERROR: In procedure scm-error:
substitute: no code for module (gcrypt hash)
building
/home/ryadav/opt/guix/20190501/store/rbx86vmir4dv036za13chqaq1f8d22ml-sqlite-autoconf-3260000.tar.gz.drv...
building
/home/ryadav/opt/guix/20190501/store/f75pwpvnxcf5fa10i42virypan2calan-bzip2-1.0.6.tar.gz.drv...
building
/home/ryadav/opt/guix/20190501/store/yf3llj2jdr0skb0zvpw5anwkdgwrfp2s-coreutils-8.30.tar.xz.drv...
building
/home/ryadav/opt/guix/20190501/store/agl7r65awykgsx3xbvfsh33y393am3bn-file-5.33.tar.gz.drv...
building
/home/ryadav/opt/guix/20190501/store/agl7r65awykgsx3xbvfsh33y393am3bn-file-5.33.tar.gz.drv...
-builder for
`/home/ryadav/opt/guix/20190501/store/f75pwpvnxcf5fa10i42virypan2calan-bzip2-1.0.6.tar.gz.drv'
failed with exit code 1
build of
/home/ryadav/opt/guix/20190501/store/f75pwpvnxcf5fa10i42virypan2calan-bzip2-1.0.6.tar.gz.drv
failed
View build log at
'/home/ryadav/opt/guix/20190501/var/log/guix/drvs/f7/5pwpvnxcf5fa10i42virypan2calan-bzip2-1.0.6.tar.gz.drv.bz2'.
builder for
`/home/ryadav/opt/guix/20190501/store/yf3llj2jdr0skb0zvpw5anwkdgwrfp2s-coreutils-8.30.tar.xz.drv'
failed with exit code 1
build of
/home/ryadav/opt/guix/20190501/store/yf3llj2jdr0skb0zvpw5anwkdgwrfp2s-coreutils-8.30.tar.xz.drv
failed
View build log at
'/home/ryadav/opt/guix/20190501/var/log/guix/drvs/yf/3llj2jdr0skb0zvpw5anwkdgwrfp2s-coreutils-8.30.tar.xz.drv.bz2'.
builder for
`/home/ryadav/opt/guix/20190501/store/agl7r65awykgsx3xbvfsh33y393am3bn-file-5.33.tar.gz.drv'
failed with exit code 1
build of
/home/ryadav/opt/guix/20190501/store/agl7r65awykgsx3xbvfsh33y393am3bn-file-5.33.tar.gz.drv
failed
View build log at
'/home/ryadav/opt/guix/20190501/var/log/guix/drvs/ag/l7r65awykgsx3xbvfsh33y393am3bn-file-5.33.tar.gz.drv.bz2'.
builder for
`/home/ryadav/opt/guix/20190501/store/rbx86vmir4dv036za13chqaq1f8d22ml-sqlite-autoconf-3260000.tar.gz.drv'
failed with exit code 1
build of
/home/ryadav/opt/guix/20190501/store/rbx86vmir4dv036za13chqaq1f8d22ml-sqlite-autoconf-3260000.tar.gz.drv
failed
View build log at
'/home/ryadav/opt/guix/20190501/var/log/guix/drvs/rb/x86vmir4dv036za13chqaq1f8d22ml-sqlite-autoconf-3260000.tar.gz.drv.bz2'.
cannot build derivation
`/home/ryadav/opt/guix/20190501/store/i31k6qjqqw0hnkd1xnwh0410z9x1hf35-sqlite-3.26.0.drv':
1 dependencies couldn't be built
guix package: error: build of
`/home/ryadav/opt/guix/20190501/store/i31k6qjqqw0hnkd1xnwh0410z9x1hf35-sqlite-3.26.0.drv'
failed
*#+end_log*

[-- Attachment #2: Type: text/html, Size: 12385 bytes --]

Re: [Non-root Guix]: Unable to build guix

[Ricardo Wurmus]

Hi,

> I am trying to prepare guix for installing at non-root location. I have
> followed the documentation available here:
> https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#error-in-procedure-scm-error-no-code-for-module-gcrypt-hash

I’d just like to note that this is not official documentation of the
Guix project.  I don’t know if this is still accurate information.

Do you want to build everything from source on your system or do you
prefer to use binaries from the Guix project’s build farms?  Do you have
access to user namespaces / can you use “unshare” for file system
shenanigans?

> ./pre-inst-env guix package -i guix

We don’t recommend using Guix from a git checkout in most cases, because…

> substitute: ERROR: In procedure scm-error:
> substitute: no code for module (gcrypt hash)

…you would need to take care of providing the appropriate Guile
environment all by yourself whereas “guix pull” does all of this for
you.  This error tells you that guile-gcrypt is not available in the
environment in which the daemon runs.

I would also like to suggest an alternative to a root-less installation.
At the HPC cluster where I work we run the guix-daemon as root on a
virtual machine where /gnu/store is a writable NFS share.  Every machine
that uses Guix only needs to mount this share as read-only.

--
Ricardo

Re: [Non-root Guix]: Unable to build guix

[Ricardo Wurmus]

rohit yadav <rohityadav@utexas.edu> writes:

> I am trying to prepare guix for installing at non-root location. I have
> followed the documentation available here:
> https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#error-in-procedure-scm-error-no-code-for-module-gcrypt-hash

I forgot to say that we discussed the various non-root installation
methods on the Guix HPC blog:

https://guix-hpc.bordeaux.inria.fr/blog/2017/10/using-guix-without-being-root/

I hope you’ll find this useful.

--
Ricardo

Re: [Non-root Guix]: Unable to build guix

[rohit yadav]

[-- Attachment #1: Type: text/plain, Size: 3051 bytes --]

Hi Ricardo,



On Thu, May 2, 2019 at 3:33 PM Ricardo Wurmus <rekado@elephly.net> wrote:

>
> Hi,
>
> > I am trying to prepare guix for installing at non-root location. I have
> > followed the documentation available here:
> >
> https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#error-in-procedure-scm-error-no-code-for-module-gcrypt-hash
>
> I’d just like to note that this is not official documentation of the
> Guix project.  I don’t know if this is still accurate information.
>
> I understand that this is not an official documentation. I installed guix
through office documentation on my system. It is working fine.


> Do you want to build everything from source on your system or do you
> prefer to use binaries from the Guix project’s build farms?  Do you have
> access to user namespaces / can you use “unshare” for file system
> shenanigans?
>
I need to build guix for non-root location because at several places its
hard to convince to use such system. However, I need several packages which
are not usually available or outdated. The guix projects allows me to do
so. To accomplish that I need truly non-root permission oriented method to
bootstrap and install whatever I deem necessary for my use case.
Unfortunately, I am stuck at Centos6 which uses old
kernel 2.6.32-696.el6.x86_64. I am able to install nix on this system at
non-root location. It works fine but NIx community has decided to use 2.0
which only works on newer kernel. Therefore, I have to now backport several
packages. Making it really hard because lot of packages are broken. Its
manageable but not ideal. Whereas the guix seems to be independent of
kernel (atleast that is my assumption so far).

>
> > ./pre-inst-env guix package -i guix
>
> We don’t recommend using Guix from a git checkout in most cases, because…
>
> > substitute: ERROR: In procedure scm-error:
> > substitute: no code for module (gcrypt hash)
>

I understand but that is why my personal work machine (running Ubuntu
18.04) is using the 1.0.0 guix not the latest git repo one. Also, I am
boostraping guix/gnu store from tag v1.0.0.

>
> …you would need to take care of providing the appropriate Guile
> environment all by yourself whereas “guix pull” does all of this for
> you.  This error tells you that guile-gcrypt is not available in the
> environment in which the daemon runs.
>

This is the place where I am struggling. I am not sure what more do I need
to do then described in pjotrp notes. I could not find anything specific in
the guix documentation as well to accomplish this.

I would also like to suggest an alternative to a root-less installation.
> At the HPC cluster where I work we run the guix-daemon as root on a
> virtual machine where /gnu/store is a writable NFS share.  Every machine
> that uses Guix only needs to mount this share as read-only.
>

This option is not available as I stated earlier, I do not have the root
permission.

>
> --
> Ricardo
>
> --Rohit

[-- Attachment #2: Type: text/html, Size: 5866 bytes --]

Re: [Non-root Guix]: Unable to build guix

[rohit yadav]

[-- Attachment #1: Type: text/plain, Size: 1175 bytes --]

Ricardo,

Thanks for pointing that out. I have gone through that. However, none of
those options are applicable here. All of those at some point requires root
permission. The proot is the only option but it is terribly slow and also
several time does not work with dispatching jobs etc (do not have real
example but had felt difficulty beyond slow speed).

For now I am managing with Nix but I need something that is portable to
older kernel. Unfortunately, at several work places that is the case.

Regards,
Rohit

On Thu, May 2, 2019 at 3:35 PM Ricardo Wurmus <rekado@elephly.net> wrote:

>
> rohit yadav <rohityadav@utexas.edu> writes:
>
> > I am trying to prepare guix for installing at non-root location. I have
> > followed the documentation available here:
> >
> https://github.com/pjotrp/guix-notes/blob/master/INSTALL.org#error-in-procedure-scm-error-no-code-for-module-gcrypt-hash
>
> I forgot to say that we discussed the various non-root installation
> methods on the Guix HPC blog:
>
>
> https://guix-hpc.bordeaux.inria.fr/blog/2017/10/using-guix-without-being-root/
>
> I hope you’ll find this useful.
>
> --
> Ricardo
>
>

[-- Attachment #2: Type: text/html, Size: 2722 bytes --]

Re: [Non-root Guix]: Unable to build guix

[Ricardo Wurmus]

Hi Rohit,

> Unfortunately, I am stuck at Centos6 which uses old
> kernel 2.6.32-696.el6.x86_64. I am able to install nix on this system at
> non-root location. It works fine but NIx community has decided to use 2.0
> which only works on newer kernel. Therefore, I have to now backport several
> packages. Making it really hard because lot of packages are broken. Its
> manageable but not ideal. Whereas the guix seems to be independent of
> kernel (atleast that is my assumption so far).

I feel your pain.  I still need to sometimes touch CentOS 6 systems.
Guix is largely kernel-independent but the GNU C library is not.  It has
a minimum kernel requirement of 3.x (I don’t know the exact version).

We have patched it to accept the heavily patched RHEL 6 kernel, because
it does seem to provide backports of all the features the GNU C library
depends on.  You may run into some problems with some packages, though.
(I think there were problems with JDK and Qt, and I’m not sure if we
managed to completely work around them.)

> I would also like to suggest an alternative to a root-less installation.
>> At the HPC cluster where I work we run the guix-daemon as root on a
>> virtual machine where /gnu/store is a writable NFS share.  Every machine
>> that uses Guix only needs to mount this share as read-only.
>>
>
> This option is not available as I stated earlier, I do not have the root
> permission.

That’s a pity!

--
Ricardo

Re: [Non-root Guix]: Unable to build guix

[Ricardo Wurmus]

rohit yadav <rohityadav@utexas.edu> writes:

>> Do you want to build everything from source on your system or do you
>> prefer to use binaries from the Guix project’s build farms?  Do you have
>> access to user namespaces / can you use “unshare” for file system
>> shenanigans?
>>
> I need to build guix for non-root location because at several places its
> hard to convince to use such system. However, I need several packages which
> are not usually available or outdated. The guix projects allows me to do
> so. To accomplish that I need truly non-root permission oriented method to
> bootstrap and install whatever I deem necessary for my use case.

I see.  Unfortunately you will end up having to compile everything from
source, C library, GCC,… — all of it.  When using a different store
location it is impossible to use pre-built binaries, unfortunately.

>> …you would need to take care of providing the appropriate Guile
>> environment all by yourself whereas “guix pull” does all of this for
>> you.  This error tells you that guile-gcrypt is not available in the
>> environment in which the daemon runs.
>>
>
> This is the place where I am struggling. I am not sure what more do I need
> to do then described in pjotrp notes. I could not find anything specific in
> the guix documentation as well to accomplish this.

There are two ways I can think of: the first is to use the slow
proot-style Guix to run “guix environment guix”; from within that
environment you will have all you need to build Guix.  You can use that
to compile Guix with a different store prefix and later use *that* Guix
to run “guix environment guix”.  Eventually, you could drop the
proot-style Guix.

I’m not sure it’s worth doing.  It seems like a lot of work and you will
never be able to benefit from pre-built binaries :-/

The other way is to manually build “guile-gcrypt”, using your system
compiler toolchain.  The sources are available here:
https://notabug.org/cwebber/guile-gcrypt

…

I just thought of another way: how about running the Guix daemon in a
virtual machine (you can download a ready-made VM image from the project
website) and telling Guix to talk to the VM over the network?  Running
the installed software unfortunately would have to go through the VM,
but this might be another option depending on your circumstances.

--
Ricardo

Re: [Non-root Guix]: Unable to build guix

[rohit yadav]

[-- Attachment #1: Type: text/plain, Size: 2033 bytes --]

On Thu, May 2, 2019 at 4:37 PM Ricardo Wurmus <rekado@elephly.net> wrote:

>
> Hi Rohit,
>
> > Unfortunately, I am stuck at Centos6 which uses old
> > kernel 2.6.32-696.el6.x86_64. I am able to install nix on this system at
> > non-root location. It works fine but NIx community has decided to use 2.0
> > which only works on newer kernel. Therefore, I have to now backport
> several
> > packages. Making it really hard because lot of packages are broken. Its
> > manageable but not ideal. Whereas the guix seems to be independent of
> > kernel (atleast that is my assumption so far).
>
> I feel your pain.  I still need to sometimes touch CentOS 6 systems.
> Guix is largely kernel-independent but the GNU C library is not.  It has
> a minimum kernel requirement of 3.x (I don’t know the exact version).
>
> We have patched it to accept the heavily patched RHEL 6 kernel, because
> it does seem to provide backports of all the features the GNU C library
> depends on.  You may run into some problems with some packages, though.
> (I think there were problems with JDK and Qt, and I’m not sure if we
> managed to completely work around them.)
>
The JDK is easy to get around so I am fine there. I rarely using any
graphical softwares, so I may be fine here. Its mostly for the purpose of
computing software which I want on the HPC, other personal software I can
work on my machine. But thanks for pointing this out. Learning about lower
level details about the kernel and its dependency has been on my list but
never been able to find time.

>
> > I would also like to suggest an alternative to a root-less installation.
> >> At the HPC cluster where I work we run the guix-daemon as root on a
> >> virtual machine where /gnu/store is a writable NFS share.  Every machine
> >> that uses Guix only needs to mount this share as read-only.
> >>
> >
> > This option is not available as I stated earlier, I do not have the root

> permission.
>
> That’s a pity!


> --
> Ricardo
>
>

[-- Attachment #2: Type: text/html, Size: 3365 bytes --]

Re: [Non-root Guix]: Unable to build guix

[rohit yadav]

[-- Attachment #1: Type: text/plain, Size: 3463 bytes --]

On Thu, May 2, 2019 at 4:50 PM Ricardo Wurmus <rekado@elephly.net> wrote:

>
> rohit yadav <rohityadav@utexas.edu> writes:
>
> >> Do you want to build everything from source on your system or do you
> >> prefer to use binaries from the Guix project’s build farms?  Do you have
> >> access to user namespaces / can you use “unshare” for file system
> >> shenanigans?
> >>
> > I need to build guix for non-root location because at several places its
> > hard to convince to use such system. However, I need several packages
> which
> > are not usually available or outdated. The guix projects allows me to do
> > so. To accomplish that I need truly non-root permission oriented method
> to
> > bootstrap and install whatever I deem necessary for my use case.
>
> I see.  Unfortunately you will end up having to compile everything from
> source, C library, GCC,… — all of it.  When using a different store
> location it is impossible to use pre-built binaries, unfortunately.
>
Yes, I am fine with this. Its actually not so bad. Most of the time its
only the beginning that little time (few days) needs to be invested but
returns are totally worth it. Also, this way I am able to find out the
broken packages as well which usually do not get reflected till a binary
exists in the store.

>
> >> …you would need to take care of providing the appropriate Guile
> >> environment all by yourself whereas “guix pull” does all of this for
> >> you.  This error tells you that guile-gcrypt is not available in the
> >> environment in which the daemon runs.
> >>
> >
> > This is the place where I am struggling. I am not sure what more do I
> need
> > to do then described in pjotrp notes. I could not find anything specific
> in
> > the guix documentation as well to accomplish this.
>
> There are two ways I can think of: the first is to use the slow
> proot-style Guix to run “guix environment guix”; from within that
> environment you will have all you need to build Guix.  You can use that
> to compile Guix with a different store prefix and later use *that* Guix
> to run “guix environment guix”.  Eventually, you could drop the
> proot-style Guix.
>
> I’m not sure it’s worth doing.  It seems like a lot of work and you will
> never be able to benefit from pre-built binaries :-/
>
> The other way is to manually build “guile-gcrypt”, using your system
> compiler toolchain.  The sources are available here:
> https://notabug.org/cwebber/guile-gcrypt
>
> …
>
> I just thought of another way: how about running the Guix daemon in a
> virtual machine (you can download a ready-made VM image from the project
> website) and telling Guix to talk to the VM over the network?  Running
> the installed software unfortunately would have to go through the VM,
> but this might be another option depending on your circumstances.
>
> I am not sure about this option. At the end of the day I need packages
compiled for a specific kernel version to a non-root location. So having a
VM with guix installed would not be so useful. I am just stuck at the older
glibc. I will manage with whatever I have using Nix 17.03 release which
seems to still work but for my home machine I will continue to explore
guix.

Thanks again for quick response. I now understand where I stand what are
the actual problems. We can close this thread now.

> --
> Ricardo
>

--Rohit
--Rohit

[-- Attachment #2: Type: text/html, Size: 5112 bytes --]

Re: [Non-root Guix]: Unable to build guix

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 652 bytes --]

Ricardo Wurmus <rekado@elephly.net> writes:

> When using a different store
> location it is impossible to use pre-built binaries, unfortunately.

Nitpick: it is possible to use pre-built binaries when using a custom
store location, if the substitutes were built for the same store
location.  But since the build farm only builds for the "/gnu/store"
location, then yes, in practice anyone who uses a different store
location will not be able to get substitutes from the build farm.

I know Ricardo already knows this; I just wanted to clarify this in case
anyone else was reading and might have gotten the wrong impression.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: [Non-root Guix]: Unable to build guix

[Mark H Weaver]

Hi Ricardo,

Ricardo Wurmus <rekado@elephly.net> writes:

> rohit yadav <rohityadav@utexas.edu> writes:
>
>>> Do you want to build everything from source on your system or do you
>>> prefer to use binaries from the Guix project’s build farms?  Do you have
>>> access to user namespaces / can you use “unshare” for file system
>>> shenanigans?
>>>
>> I need to build guix for non-root location because at several places its
>> hard to convince to use such system. However, I need several packages which
>> are not usually available or outdated. The guix projects allows me to do
>> so. To accomplish that I need truly non-root permission oriented method to
>> bootstrap and install whatever I deem necessary for my use case.
>
> I see.  Unfortunately you will end up having to compile everything from
> source, C library, GCC,… — all of it.  When using a different store
> location it is impossible to use pre-built binaries, unfortunately.

If one is able to obtain write access to any directory accessible via an
absolute path name of no more than 10 bytes, e.g. "/tmp/xxxxx",
"/var/tmp/x", "/home/xx/x", or possibly even "/home/xxxx" or
"/home/xxx", then it may be possible to avoid compiling everything from
source code.

In principle, I believe it should be possible to rewrite all occurrences
of "/gnu/store" in our store outputs to another string of the same
length, or to a shorter string by adding redundant slashes.  Note that
one must take care to rewrite the targets of symbolic links as well.

Our grafting system already crucially depends on being able to reliably
find and rewrite store references, and therefore the Guix project is
naturally compelled to ensure that our package outputs encode store
references as plain byte strings.  For example, although we generally
prefer to minimize changes to upstream packages, we modified our C
compiler to avoid encoding string literals within immediate operands in
generated machine code, as upstream GCC will sometimes do on x86_64.

In summary, Guix has already laid the groundwork to support this kind of
rewriting of binaries, and we could put it to good use here.  I suggest
adding a mechanism to our offloading code that allows rewriting the
store prefix within downloaded binaries to a local store prefix of equal
or lesser length.

What do you think?

      Mark

Re: [Non-root Guix]: Unable to build guix

[Ricardo Wurmus]

Mark H Weaver <mhw@netris.org> writes:
[…]
> Ricardo Wurmus <rekado@elephly.net> writes:
[…]
>> I see.  Unfortunately you will end up having to compile everything from
>> source, C library, GCC,… — all of it.  When using a different store
>> location it is impossible to use pre-built binaries, unfortunately.
>
> If one is able to obtain write access to any directory accessible via an
> absolute path name of no more than 10 bytes, e.g. "/tmp/xxxxx",
> "/var/tmp/x", "/home/xx/x", or possibly even "/home/xxxx" or
> "/home/xxx", then it may be possible to avoid compiling everything from
> source code.

I think it’s worth supporting prefix rewrites. On a system where the
user does not have root access and no user namespaces the daemon will
not be able to build anything in isolation.  The best case here is to
*only* use substitutes and to limit the local operations to rewriting
the prefix.

This requires modifying store items before unpacking them.  Since root
is not involved this should only ever affect one user.

--
Ricardo