How should I install non-guix software?

How should I install non-guix software?

[Luther Thompson]

I don't know if this has been discussed before, but is there a best
practice for installing software from outside the Guix package system?
If I have a source tarball for a program that doesn't have a Guix
package, or if I write my own program, there just isn't any good place
to put the installed files. Scripts have the additional problem of not
being able to use their shebang line.

As a more unique example, I was recently using a React tutorial. IIRC,
I ran the command `npm start`. It tried to run another installed script.
I had to manually change the shebang line of that other script to point
to /home/luther/.guix-profile/bin/node to get it to run.

I can think of a few solutions, all of them difficult:

1. Write a Guix package for every little piece of software we want to
run. I have yet to successfully do this, but I might decide to learn
someday. I already know Scheme. AFAIK, this won't work for node
packages, but it should work for everything else. This solution would
be a lot of work for users.

2. Put ~/bin in our $PATH and install all our executables there. This
wouldn't work for the majority of software that installs more than just
executable files.

3. Instead of installing the software, run it directly from whatever
directory we unpacked/compiled it to.

Unless we use solution 1, we also need a solution for scripts.

1. Make a soft link at /usr/bin/env in the root filesystem pointing
to /run/current-system/profile/bin/env. This seems like the simplest
solution, but I figure there must be some reason the devs haven't
already done this.

2. Know which interpreter we need for each program and invoke that
interpreter explicitly.

3. Manually change the shebang line for every script to point into
our .guix-profile. I don't like this idea, because it would mean
hard-coding the name of our home directory into every script. We can't
really do that with programs that we want to hack on and publish.

Currently, I invoke binaries from their own directories and for
scripts, I invoke the interpreters. I've only been using GuixSD for a
short time, so I doubt that this is a fool-proof solution. How do
you guys handle this?

Luther

Re: How should I install non-guix software?

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 9841 bytes --]

Hi Luther,

Those are really good questions.  I'll try to explain what I know, and
hopefully it'll be useful to you.

Luther Thompson <lutheroto@gmail.com> writes:

> I don't know if this has been discussed before, but is there a best
> practice for installing software from outside the Guix package system?

For custom packages, you can define your own via the GUIX_PACKAGE_PATH.
See "Package Modules" in the Guix manual for how to do that [1].

For software installed via other package management tools like npm,
Maven, or pip, it's trickier.  Those package managers ought to work out
of the box.  I use Maven (installed via Guix) frequently on a GuixSD
system, and it's always worked well for me.

However, because (1) those package managers don't provide the same kind
of strong guarantees about dependencies that Guix does, and (2) because
GuixSD doesn't follow the FHS (for good reasons), there is definitely
the risk that some software you install via those other package managers
won't work correctly out of the box on GuixSD.

> If I have a source tarball for a program that doesn't have a Guix
> package, or if I write my own program, there just isn't any good place
> to put the installed files. Scripts have the additional problem of not
> being able to use their shebang line.

Yes, the hard-coded references are a common problem.  It's always been a
problem when software assumes that the components they require will
exist at certain absolute paths, since even among systems that don't use
Guix these paths can vary.  Software that has been designed from the
start to be portable will sometimes mitigate this problem by providing a
way for users to tell the program where its dependencies live (e.g., via
an option to the ./configure script).  However, this is still a common
problem.

I know of at least a few ways to deal with this problem.  They are:

1) Work with upstream to provide a way for users to specify the
   location.  For example, Pierre is doing this for magit right now [2].

2) Replace every hard-coded occurrence of the path with a correct path.
   We do this all the time.  For just one example, refer to the use of
   the substitute* procedure in the package definition for pies (run
   "guix edit pies").

3) Run the software in an environment where its expectations are met.

For (3), like you suggested, one way to do it is to create symlinks in
your system.  That's fine for a quick, local solution, but it isn't
appropriate for software that will be included in Guix proper.

I've sometimes created "/bin/bash" and "/usr/bin/env" on my system for
reasons like this.  When using GuixSD, you can define these kinds of
files in your OS configuration using the special-files-service-type.
See "Base Services" in the manual for details [3].

> As a more unique example, I was recently using a React tutorial. IIRC,
> I ran the command `npm start`. It tried to run another installed script.
> I had to manually change the shebang line of that other script to point
> to /home/luther/.guix-profile/bin/node to get it to run.

The build systems of Guix (e.g., gnu-build-system) generally have a
build phase that automatically finds and patches shebang lines,
replacing them with absolute paths to the corresponding interpreters in
the store.  Unfortunately, if other package managers like npm, Maven,
pip, etc. do not provide a way to change hard-coded paths - and I don't
think any of them do - this will always be a problem.  In my opinion, if
you have to use these kinds of tools to install software on GuixSD, the
best compromise is probably to create symlinks in your system as above.

As far as I know, there isn't a good, holistic way in GuixSD to deal
with hard-coded paths in software that has been installed via other
package managers like npm.  If someone knows better, I'd love to hear
otherwise!

> 1. Write a Guix package for every little piece of software we want to
> run. I have yet to successfully do this, but I might decide to learn
> someday. I already know Scheme. AFAIK, this won't work for node
> packages, but it should work for everything else. This solution would
> be a lot of work for users.

In some ways, this is the best approach.  However, I know it's easier
said than done.  If you're ever feeling stuck, you can ask for help here
as long as the software you're trying to package is free software.  It's
also helpful to look at the thousands of existing examples in the Guix
source tree, which are located in the gnu/packages directory.

> 2. Put ~/bin in our $PATH and install all our executables there. This
> wouldn't work for the majority of software that installs more than just
> executable files.

I'm not sure what you mean here.  How would putting ~/bin in your $PATH
enable software that hard-codes a path of "/bin/bash" to find the bash
program?

> 3. Instead of installing the software, run it directly from whatever
> directory we unpacked/compiled it to.

Wouldn't this also fail to address the problem of hard-coded paths?

> 1. Make a soft link at /usr/bin/env in the root filesystem pointing
> to /run/current-system/profile/bin/env. This seems like the simplest
> solution, but I figure there must be some reason the devs haven't
> already done this.

Yeah, I think creating symlinks like this is probably the easiest way to
get "foreign" software working on a GuixSD system.  It doesn't always
work, but when it does, it's an easy hack.

However, like I said, that solution isn't appropriate for software that
will be included in Guix proper.  This is because it just isn't
necessary: if the software is packaged in Guix proper, then all of its
dependencies are guaranteed to exist (at exactly the intended versions)
in the store.

In fact, if we added symlinks like "/bin/bash" to GuixSD by default, it
would mask certain kinds of packaging problems.  If I packaged a piece
of software that uses the hard-coded path /bin/bash at runtime, it would
run fine on my system because I have a "/bin/bash" symlink, which points
to some (indeterminate) version of bash.  When you install the same
package, your /bin/bash symlink might point to a different version of
bash, so the software might behave differently for you than it does for
me.  In addition, if for example you make a bundle of the software using
"guix pack" and ship it to a system that doesn't have /bin/bash, the
software might break entirely.

My understanding is that one of the reasons why Guix doesn't follow the
FHS is to make it difficult to accidentally package software that will
break in these sorts of ways.  If /bin/bash is missing from my system
when I package software that assumes it will exist at that path, I will
be more likely to notice that the software depends on something outside
the store.  I can then fix it by replacing the path with an exact
version of bash that is managed by Guix.  In this way, the problems
mentioned above will not happen, since whenever somebody installs the
software via Guix, it is guaranteed that the installed software will use
the exact same version of bash that I specified in the package
definition (assuming that the user isn't using a different version of
Guix)

In this way, GuixSD is designed to be a "pure" system that tries hard to
prevent undeclared dependencies from creeping into your environment.
Undeclared dependencies are bad because they can interfere with or break
software.  The usual FHS runs counter to this goal, since it encourages
software to make specific assumptions about the environment.  With Guix
(and Nix), it is no longer necessary to make those kinds of assumptions
in the first place, since all the dependencies are always known exactly.

> 2. Know which interpreter we need for each program and invoke that
> interpreter explicitly.

Yes, this is also an option, but as you might imagine, it doesn't scale
past a handful of scripts/programs.

> 3. Manually change the shebang line for every script to point into
> our .guix-profile. I don't like this idea, because it would mean
> hard-coding the name of our home directory into every script. We can't
> really do that with programs that we want to hack on and publish.

Yeah, I agree that isn't a great solution, either.  But it will work in
a pinch, too.

> Currently, I invoke binaries from their own directories and for
> scripts, I invoke the interpreters. I've only been using GuixSD for a
> short time, so I doubt that this is a fool-proof solution. How do
> you guys handle this?

In sum, I guess the most common things that I do are (in no particular
order):

* Create symlinks in my system when needed.

* Manually patch specific files (e.g., replacing their shebang lines, or
  using patchelf to replace the ld interpreter path).

* Create package definitions either in my GUIX_PACKAGE_PATH or in a
  special guix.scm file (which you can install into a custom profile
  using "guix package -f my-file.scm").  Ultimately, if it works out, I
  upstream my packages to Guix proper so everyone can benefit.

Finally, an alternative of last resort is to run the problematic
software in a VM using a more traditional GNU/Linux distribution.
However, I didn't mention that until now, since it's basically
equivalent to giving up on trying to run it in GuixSD itself.

Anyway, I hope that helps!  Thanks for bearing with my very long reply.

Footnotes: 
[1]  https://www.gnu.org/software/guix/manual/en/html_node/Package-Modules.html#Package-Modules

[2]  https://lists.gnu.org/archive/html/help-guix/2018-08/msg00032.html

[3]  https://www.gnu.org/software/guix/manual/en/html_node/Base-Services.html#index-special_002dfiles_002dservice_002dtype

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: How should I install non-guix software?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 529 bytes --]

There is also the possibility to run you programs in a container with a mapped
folder hierarchy, e.g.:

  guix environment -C --expose=/home/foo/bar/bin=/bin -- /home/foo/qux/bin/qux

See "(guix) Invoking guix environment".

I think it's also worth mentioning the practice of including a Guix package
declaration into your source, like https://dthompson.us/projects/haunt.html
does.  It's mostly useful for development packages since stable versions should
preferably be added to Guix itself.

-- 
Pierre Neidhardt

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: How should I install non-guix software?

[Benjamin Slade]

I'm still getting to grips with Guix, so a fairly basic question which
seems connected:  how do I execute a binary? So for instance, I can
download a precompiled binary for the kitty terminal app:
https://sw.kovidgoyal.net/kitty/binary.html

But when I try to execute it in Guix, I get the message:
"./kitty: No such file or directory"

I tried symlinking /bin/bash and /usr/bin/env, but this didn't help.

I'm sure something in the manual or this thread probably answers this,
but I often find I can generalise from concrete examples better than vice-versa.

-- 
Benjamin Slade - https://babbagefiles.xyz
  `(pgp_fp: ,(21BA 2AE1 28F6 DF36 110A 0E9C A320 BBE8 2B52 EE19))
    '(sent by mu4e on Emacs running under GNU/Linux . https://gnu.org )
       `(Choose Linux ,(Choose Freedom) . https://linux.com )

Re: How should I install non-guix software?

[Nils Gillmann]

Benjamin Slade transcribed 807 bytes:
> I'm still getting to grips with Guix, so a fairly basic question which
> seems connected:  how do I execute a binary? So for instance, I can
> download a precompiled binary for the kitty terminal app:
> https://sw.kovidgoyal.net/kitty/binary.html
> 
> But when I try to execute it in Guix, I get the message:
> "./kitty: No such file or directory"
> 
> I tried symlinking /bin/bash and /usr/bin/env, but this didn't help.
> 
> I'm sure something in the manual or this thread probably answers this,
> but I often find I can generalise from concrete examples better than vice-versa.
> 
> -- 
> Benjamin Slade - https://babbagefiles.xyz
>   `(pgp_fp: ,(21BA 2AE1 28F6 DF36 110A 0E9C A320 BBE8 2B52 EE19))
>     '(sent by mu4e on Emacs running under GNU/Linux . https://gnu.org )
>        `(Choose Linux ,(Choose Freedom) . https://linux.com )
> 

In this case you have to use patchelf, and use the conventional Unix
tools to figure out what's missing in rpath and interpreter.

As far as I can guess.

Re: How should I install non-guix software?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1962 bytes --]

Indeed.

Foreign binaries must be patched to modify the interpreter (i.e. the dynamic
library loader, also known as ld-linux) and the RPATH (the library path).

$ patchelf --print-interpreter /bin/sh
/gnu/store/l4lr0f5cjd0nbsaaf8b5dmcw1a1yypr3-glibc-2.27/lib/ld-linux-x86-64.so.2
$ patchelf --set-interpreter /gnu/store/l4lr0f5cjd0nbsaaf8b5dmcw1a1yypr3-glibc-2.27/lib/ld-linux-x86-64.so.2 /path/to/foo/bin

You can then check the libraries with ldd:

$ ldd /path/to/foo/bin
	linux-vdso.so.1 (0x00007ffefb6e4000)
	libreadline.so.7 => /gnu/store/s5qvyb3lp0b12qmgiwj3754an7dr1r0s-readline-7.0.3/lib/libreadline.so.7 (0x00007f0b638f7000)
	libhistory.so.7 => /gnu/store/s5qvyb3lp0b12qmgiwj3754an7dr1r0s-readline-7.0.3/lib/libhistory.so.7 (0x00007f0b636ed000)
	libncursesw.so.6 => /gnu/store/s9gbq6q72w9vf7zjm0amjf1iw1fy856h-ncurses-6.1/lib/libncursesw.so.6 (0x00007f0b6347e000)
	libdl.so.2 => /gnu/store/l4lr0f5cjd0nbsaaf8b5dmcw1a1yypr3-glibc-2.27/lib/libdl.so.2 (0x00007f0b6327a000)
	libgcc_s.so.1 => /gnu/store/vla5j7pbkpcp39lsdfsmz7m9azn48lr4-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007f0b63063000)
	libc.so.6 => /gnu/store/l4lr0f5cjd0nbsaaf8b5dmcw1a1yypr3-glibc-2.27/lib/libc.so.6 (0x00007f0b62cb0000)
	libbgdrtm.so => not found

In this example, one library is missing.  If you check the RPATH, you can see
the list of folders that are search for libraries:

$ patchelf --print-rpath /path/to/foo/bin
/gnu/store/s5qvyb3lp0b12qmgiwj3754an7dr1r0s-readline-7.0.3/lib:/gnu/store/s9gbq6q72w9vf7zjm0amjf1iw1fy856h-ncurses-6.1/lib:/gnu/store/l4lr0f5cjd0nbsaaf8b5dmcw1a1yypr3-glibc-2.27/lib:/gnu/store/vla5j7pbkpcp39lsdfsmz7m9azn48lr4-gcc-5.5.0-lib/lib:/gnu/store/vla5j7pbkpcp39lsdfsmz7m9azn48lr4-gcc-5.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/5.5.0/../../..

You can extend the RPATH to include the missing library:

$ patchelf --set-rpath /path/to/missing/lib /path/to/foo/bin

And it should work!
Hope that helps!
-- 
Pierre Neidhardt

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: How should I install non-guix software?

[Benjamin Slade]

Thanks, all, for the extended information.

 > I would run it with strace, as in `strace -f ./kitty`. That will help
 > you find exactly which files are missing. It may be easier to just
 > build the program with Guix.

It might be, once I grok Guix better.

--
Benjamin Slade - https://babbagefiles.xyz
  `(pgp_fp: ,(21BA 2AE1 28F6 DF36 110A 0E9C A320 BBE8 2B52 EE19))
    '(sent by mu4e on Emacs running under GNU/Linux . https://gnu.org )
       `(Choose Linux ,(Choose Freedom) . https://linux.com )

Re: How should I install non-guix software?

[Luther Thompson]

Thank you for such a detailed response! Sorry it took me so long to get
back to this.

On Sat, 04 Aug 2018 21:48:41 -0700
Chris Marusich <cmmarusich@gmail.com> wrote:
> Luther Thompson <lutheroto@gmail.com> writes:  
> > If I have a source tarball for a program that doesn't have a Guix
> > package, or if I write my own program, there just isn't any good
> > place to put the installed files. Scripts have the additional
> > problem of not being able to use their shebang line.    
> 
> I've sometimes created "/bin/bash" and "/usr/bin/env" on my system for
> reasons like this.  When using GuixSD, you can define these kinds of
> files in your OS configuration using the special-files-service-type.
> See "Base Services" in the manual for details [3].  

I put `(extra-special-file "/usr/bin/env" (file-append coreutils
"/bin/env"))` in my config.scm, and it worked perfectly. That will
make things a lot easier.

> > 1. Write a Guix package for every little piece of software we want
> > to run. I have yet to successfully do this, but I might decide to
> > learn someday. I already know Scheme. AFAIK, this won't work for
> > node packages, but it should work for everything else. This
> > solution would be a lot of work for users.    
> 
> In some ways, this is the best approach.  However, I know it's easier
> said than done.  If you're ever feeling stuck, you can ask for help
> here as long as the software you're trying to package is free
> software.  It's also helpful to look at the thousands of existing
> examples in the Guix source tree, which are located in the
> gnu/packages directory.  

So it looks like this is the best way to go. Thank you for clearing
that up for me.

> > 2. Put ~/bin in our $PATH and install all our executables there.
> > This wouldn't work for the majority of software that installs more
> > than just executable files.    
> 
> I'm not sure what you mean here.  How would putting ~/bin in your
> $PATH enable software that hard-codes a path of "/bin/bash" to find
> the bash program?
>   
> > 3. Instead of installing the software, run it directly from whatever
> > directory we unpacked/compiled it to.    
> 
> Wouldn't this also fail to address the problem of hard-coded paths?  

Maybe I wasn't clear, but these points were only supposed to address
where to place installed files. I addressed shebangs as a separate
issue in the other 3 points.

> > 1. Make a soft link at /usr/bin/env in the root filesystem pointing
> > to /run/current-system/profile/bin/env. This seems like the simplest
> > solution, but I figure there must be some reason the devs haven't
> > already done this.    
> 
> In fact, if we added symlinks like "/bin/bash" to GuixSD by default,
> it would mask certain kinds of packaging problems.  If I packaged a
> piece of software that uses the hard-coded path /bin/bash at runtime,
> it would run fine on my system because I have a "/bin/bash" symlink,
> which points to some (indeterminate) version of bash.  When you
> install the same package, your /bin/bash symlink might point to a
> different version of bash, so the software might behave differently
> for you than it does for me.  In addition, if for example you make a
> bundle of the software using "guix pack" and ship it to a system that
> doesn't have /bin/bash, the software might break entirely.  

This does indeed explain why we can't have /usr/bin/env by default, but
it kind of sucks that any user-friendly configuration will have to add
it right back in.

> Anyway, I hope that helps!  Thanks for bearing with my very long
> reply.  

It was very helpful. Thank you!

> Footnotes: 
> [3]
> https://www.gnu.org/software/guix/manual/en/html_node/Base-Services.html#index-special_002dfiles_002dservice_002dtype
>   

Luther