* How should I install non-guix software? @ 2018-08-05 1:45 Luther Thompson 2018-08-05 4:48 ` Chris Marusich 0 siblings, 1 reply; 8+ messages in thread From: Luther Thompson @ 2018-08-05 1:45 UTC (permalink / raw) To: help-guix I don't know if this has been discussed before, but is there a best practice for installing software from outside the Guix package system? If I have a source tarball for a program that doesn't have a Guix package, or if I write my own program, there just isn't any good place to put the installed files. Scripts have the additional problem of not being able to use their shebang line. As a more unique example, I was recently using a React tutorial. IIRC, I ran the command `npm start`. It tried to run another installed script. I had to manually change the shebang line of that other script to point to /home/luther/.guix-profile/bin/node to get it to run. I can think of a few solutions, all of them difficult: 1. Write a Guix package for every little piece of software we want to run. I have yet to successfully do this, but I might decide to learn someday. I already know Scheme. AFAIK, this won't work for node packages, but it should work for everything else. This solution would be a lot of work for users. 2. Put ~/bin in our $PATH and install all our executables there. This wouldn't work for the majority of software that installs more than just executable files. 3. Instead of installing the software, run it directly from whatever directory we unpacked/compiled it to. Unless we use solution 1, we also need a solution for scripts. 1. Make a soft link at /usr/bin/env in the root filesystem pointing to /run/current-system/profile/bin/env. This seems like the simplest solution, but I figure there must be some reason the devs haven't already done this. 2. Know which interpreter we need for each program and invoke that interpreter explicitly. 3. Manually change the shebang line for every script to point into our .guix-profile. I don't like this idea, because it would mean hard-coding the name of our home directory into every script. We can't really do that with programs that we want to hack on and publish. Currently, I invoke binaries from their own directories and for scripts, I invoke the interpreters. I've only been using GuixSD for a short time, so I doubt that this is a fool-proof solution. How do you guys handle this? Luther ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: How should I install non-guix software? 2018-08-05 1:45 How should I install non-guix software? Luther Thompson @ 2018-08-05 4:48 ` Chris Marusich 2018-08-05 8:58 ` Pierre Neidhardt 2018-08-13 0:53 ` Luther Thompson 0 siblings, 2 replies; 8+ messages in thread From: Chris Marusich @ 2018-08-05 4:48 UTC (permalink / raw) To: Luther Thompson; +Cc: help-guix [-- Attachment #1: Type: text/plain, Size: 9841 bytes --] Hi Luther, Those are really good questions. I'll try to explain what I know, and hopefully it'll be useful to you. Luther Thompson <lutheroto@gmail.com> writes: > I don't know if this has been discussed before, but is there a best > practice for installing software from outside the Guix package system? For custom packages, you can define your own via the GUIX_PACKAGE_PATH. See "Package Modules" in the Guix manual for how to do that [1]. For software installed via other package management tools like npm, Maven, or pip, it's trickier. Those package managers ought to work out of the box. I use Maven (installed via Guix) frequently on a GuixSD system, and it's always worked well for me. However, because (1) those package managers don't provide the same kind of strong guarantees about dependencies that Guix does, and (2) because GuixSD doesn't follow the FHS (for good reasons), there is definitely the risk that some software you install via those other package managers won't work correctly out of the box on GuixSD. > If I have a source tarball for a program that doesn't have a Guix > package, or if I write my own program, there just isn't any good place > to put the installed files. Scripts have the additional problem of not > being able to use their shebang line. Yes, the hard-coded references are a common problem. It's always been a problem when software assumes that the components they require will exist at certain absolute paths, since even among systems that don't use Guix these paths can vary. Software that has been designed from the start to be portable will sometimes mitigate this problem by providing a way for users to tell the program where its dependencies live (e.g., via an option to the ./configure script). However, this is still a common problem. I know of at least a few ways to deal with this problem. They are: 1) Work with upstream to provide a way for users to specify the location. For example, Pierre is doing this for magit right now [2]. 2) Replace every hard-coded occurrence of the path with a correct path. We do this all the time. For just one example, refer to the use of the substitute* procedure in the package definition for pies (run "guix edit pies"). 3) Run the software in an environment where its expectations are met. For (3), like you suggested, one way to do it is to create symlinks in your system. That's fine for a quick, local solution, but it isn't appropriate for software that will be included in Guix proper. I've sometimes created "/bin/bash" and "/usr/bin/env" on my system for reasons like this. When using GuixSD, you can define these kinds of files in your OS configuration using the special-files-service-type. See "Base Services" in the manual for details [3]. > As a more unique example, I was recently using a React tutorial. IIRC, > I ran the command `npm start`. It tried to run another installed script. > I had to manually change the shebang line of that other script to point > to /home/luther/.guix-profile/bin/node to get it to run. The build systems of Guix (e.g., gnu-build-system) generally have a build phase that automatically finds and patches shebang lines, replacing them with absolute paths to the corresponding interpreters in the store. Unfortunately, if other package managers like npm, Maven, pip, etc. do not provide a way to change hard-coded paths - and I don't think any of them do - this will always be a problem. In my opinion, if you have to use these kinds of tools to install software on GuixSD, the best compromise is probably to create symlinks in your system as above. As far as I know, there isn't a good, holistic way in GuixSD to deal with hard-coded paths in software that has been installed via other package managers like npm. If someone knows better, I'd love to hear otherwise! > 1. Write a Guix package for every little piece of software we want to > run. I have yet to successfully do this, but I might decide to learn > someday. I already know Scheme. AFAIK, this won't work for node > packages, but it should work for everything else. This solution would > be a lot of work for users. In some ways, this is the best approach. However, I know it's easier said than done. If you're ever feeling stuck, you can ask for help here as long as the software you're trying to package is free software. It's also helpful to look at the thousands of existing examples in the Guix source tree, which are located in the gnu/packages directory. > 2. Put ~/bin in our $PATH and install all our executables there. This > wouldn't work for the majority of software that installs more than just > executable files. I'm not sure what you mean here. How would putting ~/bin in your $PATH enable software that hard-codes a path of "/bin/bash" to find the bash program? > 3. Instead of installing the software, run it directly from whatever > directory we unpacked/compiled it to. Wouldn't this also fail to address the problem of hard-coded paths? > 1. Make a soft link at /usr/bin/env in the root filesystem pointing > to /run/current-system/profile/bin/env. This seems like the simplest > solution, but I figure there must be some reason the devs haven't > already done this. Yeah, I think creating symlinks like this is probably the easiest way to get "foreign" software working on a GuixSD system. It doesn't always work, but when it does, it's an easy hack. However, like I said, that solution isn't appropriate for software that will be included in Guix proper. This is because it just isn't necessary: if the software is packaged in Guix proper, then all of its dependencies are guaranteed to exist (at exactly the intended versions) in the store. In fact, if we added symlinks like "/bin/bash" to GuixSD by default, it would mask certain kinds of packaging problems. If I packaged a piece of software that uses the hard-coded path /bin/bash at runtime, it would run fine on my system because I have a "/bin/bash" symlink, which points to some (indeterminate) version of bash. When you install the same package, your /bin/bash symlink might point to a different version of bash, so the software might behave differently for you than it does for me. In addition, if for example you make a bundle of the software using "guix pack" and ship it to a system that doesn't have /bin/bash, the software might break entirely. My understanding is that one of the reasons why Guix doesn't follow the FHS is to make it difficult to accidentally package software that will break in these sorts of ways. If /bin/bash is missing from my system when I package software that assumes it will exist at that path, I will be more likely to notice that the software depends on something outside the store. I can then fix it by replacing the path with an exact version of bash that is managed by Guix. In this way, the problems mentioned above will not happen, since whenever somebody installs the software via Guix, it is guaranteed that the installed software will use the exact same version of bash that I specified in the package definition (assuming that the user isn't using a different version of Guix) In this way, GuixSD is designed to be a "pure" system that tries hard to prevent undeclared dependencies from creeping into your environment. Undeclared dependencies are bad because they can interfere with or break software. The usual FHS runs counter to this goal, since it encourages software to make specific assumptions about the environment. With Guix (and Nix), it is no longer necessary to make those kinds of assumptions in the first place, since all the dependencies are always known exactly. > 2. Know which interpreter we need for each program and invoke that > interpreter explicitly. Yes, this is also an option, but as you might imagine, it doesn't scale past a handful of scripts/programs. > 3. Manually change the shebang line for every script to point into > our .guix-profile. I don't like this idea, because it would mean > hard-coding the name of our home directory into every script. We can't > really do that with programs that we want to hack on and publish. Yeah, I agree that isn't a great solution, either. But it will work in a pinch, too. > Currently, I invoke binaries from their own directories and for > scripts, I invoke the interpreters. I've only been using GuixSD for a > short time, so I doubt that this is a fool-proof solution. How do > you guys handle this? In sum, I guess the most common things that I do are (in no particular order): * Create symlinks in my system when needed. * Manually patch specific files (e.g., replacing their shebang lines, or using patchelf to replace the ld interpreter path). * Create package definitions either in my GUIX_PACKAGE_PATH or in a special guix.scm file (which you can install into a custom profile using "guix package -f my-file.scm"). Ultimately, if it works out, I upstream my packages to Guix proper so everyone can benefit. Finally, an alternative of last resort is to run the problematic software in a VM using a more traditional GNU/Linux distribution. However, I didn't mention that until now, since it's basically equivalent to giving up on trying to run it in GuixSD itself. Anyway, I hope that helps! Thanks for bearing with my very long reply. Footnotes: [1] https://www.gnu.org/software/guix/manual/en/html_node/Package-Modules.html#Package-Modules [2] https://lists.gnu.org/archive/html/help-guix/2018-08/msg00032.html [3] https://www.gnu.org/software/guix/manual/en/html_node/Base-Services.html#index-special_002dfiles_002dservice_002dtype -- Chris [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 832 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: How should I install non-guix software? 2018-08-05 4:48 ` Chris Marusich @ 2018-08-05 8:58 ` Pierre Neidhardt 2018-08-07 15:38 ` Benjamin Slade 2018-08-13 0:53 ` Luther Thompson 1 sibling, 1 reply; 8+ messages in thread From: Pierre Neidhardt @ 2018-08-05 8:58 UTC (permalink / raw) To: Chris Marusich; +Cc: help-guix [-- Attachment #1: Type: text/plain, Size: 529 bytes --] There is also the possibility to run you programs in a container with a mapped folder hierarchy, e.g.: guix environment -C --expose=/home/foo/bar/bin=/bin -- /home/foo/qux/bin/qux See "(guix) Invoking guix environment". I think it's also worth mentioning the practice of including a Guix package declaration into your source, like https://dthompson.us/projects/haunt.html does. It's mostly useful for development packages since stable versions should preferably be added to Guix itself. -- Pierre Neidhardt [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 487 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: How should I install non-guix software? 2018-08-05 8:58 ` Pierre Neidhardt @ 2018-08-07 15:38 ` Benjamin Slade 2018-08-07 15:42 ` Nils Gillmann 0 siblings, 1 reply; 8+ messages in thread From: Benjamin Slade @ 2018-08-07 15:38 UTC (permalink / raw) To: help-guix; +Cc: Pierre Neidhardt, Chris Marusich I'm still getting to grips with Guix, so a fairly basic question which seems connected: how do I execute a binary? So for instance, I can download a precompiled binary for the kitty terminal app: https://sw.kovidgoyal.net/kitty/binary.html But when I try to execute it in Guix, I get the message: "./kitty: No such file or directory" I tried symlinking /bin/bash and /usr/bin/env, but this didn't help. I'm sure something in the manual or this thread probably answers this, but I often find I can generalise from concrete examples better than vice-versa. -- Benjamin Slade - https://babbagefiles.xyz `(pgp_fp: ,(21BA 2AE1 28F6 DF36 110A 0E9C A320 BBE8 2B52 EE19)) '(sent by mu4e on Emacs running under GNU/Linux . https://gnu.org ) `(Choose Linux ,(Choose Freedom) . https://linux.com ) ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: How should I install non-guix software? 2018-08-07 15:38 ` Benjamin Slade @ 2018-08-07 15:42 ` Nils Gillmann 2018-08-07 17:11 ` Pierre Neidhardt [not found] ` <20180807170728.GB917@jasmine.lan> 0 siblings, 2 replies; 8+ messages in thread From: Nils Gillmann @ 2018-08-07 15:42 UTC (permalink / raw) To: Benjamin Slade; +Cc: help-guix Benjamin Slade transcribed 807 bytes: > I'm still getting to grips with Guix, so a fairly basic question which > seems connected: how do I execute a binary? So for instance, I can > download a precompiled binary for the kitty terminal app: > https://sw.kovidgoyal.net/kitty/binary.html > > But when I try to execute it in Guix, I get the message: > "./kitty: No such file or directory" > > I tried symlinking /bin/bash and /usr/bin/env, but this didn't help. > > I'm sure something in the manual or this thread probably answers this, > but I often find I can generalise from concrete examples better than vice-versa. > > -- > Benjamin Slade - https://babbagefiles.xyz > `(pgp_fp: ,(21BA 2AE1 28F6 DF36 110A 0E9C A320 BBE8 2B52 EE19)) > '(sent by mu4e on Emacs running under GNU/Linux . https://gnu.org ) > `(Choose Linux ,(Choose Freedom) . https://linux.com ) > In this case you have to use patchelf, and use the conventional Unix tools to figure out what's missing in rpath and interpreter. As far as I can guess. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: How should I install non-guix software? 2018-08-07 15:42 ` Nils Gillmann @ 2018-08-07 17:11 ` Pierre Neidhardt [not found] ` <20180807170728.GB917@jasmine.lan> 1 sibling, 0 replies; 8+ messages in thread From: Pierre Neidhardt @ 2018-08-07 17:11 UTC (permalink / raw) To: Nils Gillmann; +Cc: help-guix [-- Attachment #1: Type: text/plain, Size: 1962 bytes --] Indeed. Foreign binaries must be patched to modify the interpreter (i.e. the dynamic library loader, also known as ld-linux) and the RPATH (the library path). $ patchelf --print-interpreter /bin/sh /gnu/store/l4lr0f5cjd0nbsaaf8b5dmcw1a1yypr3-glibc-2.27/lib/ld-linux-x86-64.so.2 $ patchelf --set-interpreter /gnu/store/l4lr0f5cjd0nbsaaf8b5dmcw1a1yypr3-glibc-2.27/lib/ld-linux-x86-64.so.2 /path/to/foo/bin You can then check the libraries with ldd: $ ldd /path/to/foo/bin linux-vdso.so.1 (0x00007ffefb6e4000) libreadline.so.7 => /gnu/store/s5qvyb3lp0b12qmgiwj3754an7dr1r0s-readline-7.0.3/lib/libreadline.so.7 (0x00007f0b638f7000) libhistory.so.7 => /gnu/store/s5qvyb3lp0b12qmgiwj3754an7dr1r0s-readline-7.0.3/lib/libhistory.so.7 (0x00007f0b636ed000) libncursesw.so.6 => /gnu/store/s9gbq6q72w9vf7zjm0amjf1iw1fy856h-ncurses-6.1/lib/libncursesw.so.6 (0x00007f0b6347e000) libdl.so.2 => /gnu/store/l4lr0f5cjd0nbsaaf8b5dmcw1a1yypr3-glibc-2.27/lib/libdl.so.2 (0x00007f0b6327a000) libgcc_s.so.1 => /gnu/store/vla5j7pbkpcp39lsdfsmz7m9azn48lr4-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007f0b63063000) libc.so.6 => /gnu/store/l4lr0f5cjd0nbsaaf8b5dmcw1a1yypr3-glibc-2.27/lib/libc.so.6 (0x00007f0b62cb0000) libbgdrtm.so => not found In this example, one library is missing. If you check the RPATH, you can see the list of folders that are search for libraries: $ patchelf --print-rpath /path/to/foo/bin /gnu/store/s5qvyb3lp0b12qmgiwj3754an7dr1r0s-readline-7.0.3/lib:/gnu/store/s9gbq6q72w9vf7zjm0amjf1iw1fy856h-ncurses-6.1/lib:/gnu/store/l4lr0f5cjd0nbsaaf8b5dmcw1a1yypr3-glibc-2.27/lib:/gnu/store/vla5j7pbkpcp39lsdfsmz7m9azn48lr4-gcc-5.5.0-lib/lib:/gnu/store/vla5j7pbkpcp39lsdfsmz7m9azn48lr4-gcc-5.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/5.5.0/../../.. You can extend the RPATH to include the missing library: $ patchelf --set-rpath /path/to/missing/lib /path/to/foo/bin And it should work! Hope that helps! -- Pierre Neidhardt [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 487 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
[parent not found: <20180807170728.GB917@jasmine.lan>]
* Re: How should I install non-guix software? [not found] ` <20180807170728.GB917@jasmine.lan> @ 2018-08-07 17:22 ` Benjamin Slade 0 siblings, 0 replies; 8+ messages in thread From: Benjamin Slade @ 2018-08-07 17:22 UTC (permalink / raw) To: help-guix@gnu.org; +Cc: Nils Gillmann Thanks, all, for the extended information. > I would run it with strace, as in `strace -f ./kitty`. That will help > you find exactly which files are missing. It may be easier to just > build the program with Guix. It might be, once I grok Guix better. -- Benjamin Slade - https://babbagefiles.xyz `(pgp_fp: ,(21BA 2AE1 28F6 DF36 110A 0E9C A320 BBE8 2B52 EE19)) '(sent by mu4e on Emacs running under GNU/Linux . https://gnu.org ) `(Choose Linux ,(Choose Freedom) . https://linux.com ) ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: How should I install non-guix software? 2018-08-05 4:48 ` Chris Marusich 2018-08-05 8:58 ` Pierre Neidhardt @ 2018-08-13 0:53 ` Luther Thompson 1 sibling, 0 replies; 8+ messages in thread From: Luther Thompson @ 2018-08-13 0:53 UTC (permalink / raw) To: help-guix Thank you for such a detailed response! Sorry it took me so long to get back to this. On Sat, 04 Aug 2018 21:48:41 -0700 Chris Marusich <cmmarusich@gmail.com> wrote: > Luther Thompson <lutheroto@gmail.com> writes: > > If I have a source tarball for a program that doesn't have a Guix > > package, or if I write my own program, there just isn't any good > > place to put the installed files. Scripts have the additional > > problem of not being able to use their shebang line. > > I've sometimes created "/bin/bash" and "/usr/bin/env" on my system for > reasons like this. When using GuixSD, you can define these kinds of > files in your OS configuration using the special-files-service-type. > See "Base Services" in the manual for details [3]. I put `(extra-special-file "/usr/bin/env" (file-append coreutils "/bin/env"))` in my config.scm, and it worked perfectly. That will make things a lot easier. > > 1. Write a Guix package for every little piece of software we want > > to run. I have yet to successfully do this, but I might decide to > > learn someday. I already know Scheme. AFAIK, this won't work for > > node packages, but it should work for everything else. This > > solution would be a lot of work for users. > > In some ways, this is the best approach. However, I know it's easier > said than done. If you're ever feeling stuck, you can ask for help > here as long as the software you're trying to package is free > software. It's also helpful to look at the thousands of existing > examples in the Guix source tree, which are located in the > gnu/packages directory. So it looks like this is the best way to go. Thank you for clearing that up for me. > > 2. Put ~/bin in our $PATH and install all our executables there. > > This wouldn't work for the majority of software that installs more > > than just executable files. > > I'm not sure what you mean here. How would putting ~/bin in your > $PATH enable software that hard-codes a path of "/bin/bash" to find > the bash program? > > > 3. Instead of installing the software, run it directly from whatever > > directory we unpacked/compiled it to. > > Wouldn't this also fail to address the problem of hard-coded paths? Maybe I wasn't clear, but these points were only supposed to address where to place installed files. I addressed shebangs as a separate issue in the other 3 points. > > 1. Make a soft link at /usr/bin/env in the root filesystem pointing > > to /run/current-system/profile/bin/env. This seems like the simplest > > solution, but I figure there must be some reason the devs haven't > > already done this. > > In fact, if we added symlinks like "/bin/bash" to GuixSD by default, > it would mask certain kinds of packaging problems. If I packaged a > piece of software that uses the hard-coded path /bin/bash at runtime, > it would run fine on my system because I have a "/bin/bash" symlink, > which points to some (indeterminate) version of bash. When you > install the same package, your /bin/bash symlink might point to a > different version of bash, so the software might behave differently > for you than it does for me. In addition, if for example you make a > bundle of the software using "guix pack" and ship it to a system that > doesn't have /bin/bash, the software might break entirely. This does indeed explain why we can't have /usr/bin/env by default, but it kind of sucks that any user-friendly configuration will have to add it right back in. > Anyway, I hope that helps! Thanks for bearing with my very long > reply. It was very helpful. Thank you! > Footnotes: > [3] > https://www.gnu.org/software/guix/manual/en/html_node/Base-Services.html#index-special_002dfiles_002dservice_002dtype > Luther ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2018-08-13 0:53 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2018-08-05 1:45 How should I install non-guix software? Luther Thompson 2018-08-05 4:48 ` Chris Marusich 2018-08-05 8:58 ` Pierre Neidhardt 2018-08-07 15:38 ` Benjamin Slade 2018-08-07 15:42 ` Nils Gillmann 2018-08-07 17:11 ` Pierre Neidhardt [not found] ` <20180807170728.GB917@jasmine.lan> 2018-08-07 17:22 ` Benjamin Slade 2018-08-13 0:53 ` Luther Thompson
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).