* Reproducible Project Environments @ 2021-12-06 14:18 Peter Polidoro 2021-12-06 16:16 ` Julien Lepiller 2021-12-06 16:58 ` zimoun 0 siblings, 2 replies; 8+ messages in thread From: Peter Polidoro @ 2021-12-06 14:18 UTC (permalink / raw) To: guix-science Guix is a wonderful way to distribute software, especially for science. Thank you all for such a fantastic project and community! I am an engineer working with scientists at the Howard Hughes Medical Institute, a science philanthropy based in the United States funding researchers all over the world. I hope to help convince the researchers I work with to all use Guix someday. My question is, what files do you recommend placing within a project directory to make it as straightforward as possible to distribute that project to others to develop and use in a reproducible environment? If all of the project dependencies are in Guix as packages, then a guix.scm or manifest.scm file seems all that is needed to create a reproducible environment using guix shell, as long as someone has Guix installed on their machine. What do you recommend when some of the project dependencies are not yet Guix packages? Obviously it would be best to get those dependencies into Guix as official packages eventually, but what is the best way to handle those dependencies until then? Say that in addition to some Guix packages, the project also needs some Python packages that are available on pypi, but are not yet Guix packages. Using something like direnv, an .envrc file in the project directory can install the Guix package dependencies into a local profile, then pip install the Python dependencies into a project local virtualenv. Then the user just needs Guix and direnv installed on their machine to get a reproducible environment when they change into that directory. Specifying dependencies in an .envrc file does not seem as elegant as using a pure Guix solution, however. What is the best way to handle these dependencies without using direnv and just Guix? Should I create Guix packages for the Python dependencies in an external custom channel? If so, how do you specify custom channels when using guix shell? Is there a method that is as simple as direnv or does it involve a user separately setting up a new profile to reference the custom channel before running guix shell? Do you ever create packages within a guix.scm file or a manifest.scm file or perhaps in a separate channel.scm file in the project directory to keep the project more self contained than referencing an external custom channel? Then would you create some project local profile that references this file to use when running guix shell? Or is the recommended approach to always use a custom channel before submitting the packages to be included into Guix proper? Thanks! ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Reproducible Project Environments 2021-12-06 14:18 Reproducible Project Environments Peter Polidoro @ 2021-12-06 16:16 ` Julien Lepiller 2021-12-06 18:09 ` Peter Polidoro 2021-12-06 16:58 ` zimoun 1 sibling, 1 reply; 8+ messages in thread From: Julien Lepiller @ 2021-12-06 16:16 UTC (permalink / raw) To: guix-science, Peter Polidoro [-- Attachment #1: Type: text/plain, Size: 365 bytes --] When my projects are missing a dependency from guix, or use a specific version, I simply specify them in my guix.scm for tge project, so it's just as easy to create the environment. I remove it as soon as I contribute it to guix. I would only use a separate channel if multiple projects require the same dependency and it.s taking a lot of time to get it into guix. [-- Attachment #2: Type: text/html, Size: 406 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Reproducible Project Environments 2021-12-06 16:16 ` Julien Lepiller @ 2021-12-06 18:09 ` Peter Polidoro 2021-12-06 19:52 ` Efraim Flashner 0 siblings, 1 reply; 8+ messages in thread From: Peter Polidoro @ 2021-12-06 18:09 UTC (permalink / raw) To: Julien Lepiller; +Cc: guix-science Julien Lepiller <julien@lepiller.eu> writes: > When my projects are missing a dependency from guix, or use a > specific version, I simply specify them in my guix.scm for tge > project, so it's just as easy to create the environment. Do you have any examples you can share of using both Guix official packages and custom package specifications in a guix.scm file? Thanks! ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Reproducible Project Environments 2021-12-06 18:09 ` Peter Polidoro @ 2021-12-06 19:52 ` Efraim Flashner 0 siblings, 0 replies; 8+ messages in thread From: Efraim Flashner @ 2021-12-06 19:52 UTC (permalink / raw) To: Peter Polidoro; +Cc: Julien Lepiller, guix-science [-- Attachment #1: Type: text/plain, Size: 1033 bytes --] On Mon, Dec 06, 2021 at 01:09:23PM -0500, Peter Polidoro wrote: > > Julien Lepiller <julien@lepiller.eu> writes: > > > When my projects are missing a dependency from guix, or use a specific > > version, I simply specify them in my guix.scm for tge project, so it's > > just as easy to create the environment. > > Do you have any examples you can share of using both Guix official packages > and custom package specifications in a guix.scm file? Thanks! The most complex ones I know of are for gash-utils¹ and one I wrote for fennel². As far as a simpler one, I use this one³ for my guix.vim plugin. ¹ https://git.savannah.gnu.org/cgit/gash/gash-utils.git/tree/guix.scm ² https://git.sr.ht/~efraim/fennel/tree/main/item/guix.scm ³ https://git.sr.ht/~efraim/guix.vim/tree/master/item/guix.scm -- Efraim Flashner <efraim@flashner.co.il> רנשלפ םירפא GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Reproducible Project Environments 2021-12-06 14:18 Reproducible Project Environments Peter Polidoro 2021-12-06 16:16 ` Julien Lepiller @ 2021-12-06 16:58 ` zimoun 2021-12-06 18:13 ` Peter Polidoro 2021-12-08 14:30 ` Peter Polidoro 1 sibling, 2 replies; 8+ messages in thread From: zimoun @ 2021-12-06 16:58 UTC (permalink / raw) To: Peter Polidoro, guix-science Hi, On Mon, 06 Dec 2021 at 09:18, Peter Polidoro <peter@polidoro.io> wrote: > Guix is a wonderful way to distribute software, especially for > science. Thank you all for such a fantastic project and community! > > I am an engineer working with scientists at the Howard Hughes > Medical Institute, a science philanthropy based in the United > States funding researchers all over the world. I hope to help > convince the researchers I work with to all use Guix someday. Cool! > My question is, what files do you recommend placing within a > project directory to make it as straightforward as possible to > distribute that project to others to develop and use in a > reproducible environment? From my point of view, if we are speaking of a project gluing many thing to produce analysis, then a Git repo with: - manifest.scm containing package set for the project - channels.scm containing the revision - all your files If some package are not yet in Guix, or cannot be, they should be in a custom channel (another Git repo). All Git repos must be saved in Software Heritage. Note that channels.scm describes all the channels (official Guix and all custom channels) and each state of the channel are fixed by commit. guix time-machine -C channels.scm -- shell -m manifest.scm will do a lot of job. :-) If a Git repo disappears for some reasons, then the command would fallback to SWH, more or less. ;-) > What do you recommend when some of the project dependencies are > not yet Guix packages? Obviously it would be best to get those > dependencies into Guix as official packages eventually, but what > is the best way to handle those dependencies until then? IMHO, custom channels. https://hpc.guix.info/channels/ > Say that in addition to some Guix packages, the project also needs > some Python packages that are available on pypi, but are not yet > Guix packages. Using something like direnv, an .envrc file in the > project directory can install the Guix package dependencies into a > local profile, then pip install the Python dependencies into a > project local virtualenv. Then the user just needs Guix and direnv > installed on their machine to get a reproducible environment when > they change into that directory. Specifying dependencies in an > .envrc file does not seem as elegant as using a pure Guix > solution, however. From a reproducibility point of view, this method is “dangerous” but there is no guarantee that PyPI would be in the same state. On a side note, there is no guarantee either for transparency. For instance, https://hpc.guix.info/blog/2021/09/whats-in-a-package/ > What is the best way to handle these dependencies without using > direnv and just Guix? Should I create Guix packages for the Python > dependencies in an external custom channel? If so, how do you > specify custom channels when using guix shell? Is there a method > that is as simple as direnv or does it involve a user separately > setting up a new profile to reference the custom channel before > running guix shell? Yes, I would create a custom channel for these dependencies. Starting with “guix import pypi -r”. Yeah, it can be a lot of work; if not impossible for some stuff found in PyPI. Well, it depends on your level of transparency and the complexity of your stack. As I wrote above, guix time-machine -C channels.m -- shell -m manifest.scm appears to me the good practise. Note that manifest.scm could contain transformation or quickly define a variant using inherit. > Do you ever create packages within a guix.scm file or a > manifest.scm file or perhaps in a separate channel.scm file in the > project directory to keep the project more self contained than > referencing an external custom channel? Then would you create some > project local profile that references this file to use when > running guix shell? Or is the recommended approach to always use a > custom channel before submitting the packages to be included into > Guix proper? Thanks! To include packages to Guix proper, the package must comply with the standards. https://guix.gnu.org/manual/en/guix.html#Packaging-Guidelines If for some reasons, the package cannot be included to Guix proper, it could be included to one of these channels, https://hpc.guix.info/channels/ which avoid too many package scattering. ;-) And bonus, they often provide substitutes. Thanks for your message. Cheers, simon ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Reproducible Project Environments 2021-12-06 16:58 ` zimoun @ 2021-12-06 18:13 ` Peter Polidoro 2021-12-08 14:30 ` Peter Polidoro 1 sibling, 0 replies; 8+ messages in thread From: Peter Polidoro @ 2021-12-06 18:13 UTC (permalink / raw) To: zimoun; +Cc: guix-science zimoun <zimon.toutoune@gmail.com> writes: > > guix time-machine -C channels.scm -- shell -m manifest.scm > This seems like a nice clean way of doing it once you have a custom channel set up. > > If for some reasons, the package cannot be included to Guix > proper, it > could be included to one of these channels, > > https://hpc.guix.info/channels/ > Perhaps I will create a custom channel for our organization and then try to move packages from there into these other channels or into Guix proper once they have been tested. Thanks you! ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Reproducible Project Environments 2021-12-06 16:58 ` zimoun 2021-12-06 18:13 ` Peter Polidoro @ 2021-12-08 14:30 ` Peter Polidoro 2021-12-09 20:33 ` Ludovic Courtès 1 sibling, 1 reply; 8+ messages in thread From: Peter Polidoro @ 2021-12-08 14:30 UTC (permalink / raw) To: zimoun; +Cc: guix-science > > From my point of view, if we are speaking of a project gluing > many thing > to produce analysis, then a Git repo with: > > - manifest.scm containing package set for the project > - channels.scm containing the revision > - all your files Does 'guix describe - f channels' always return a list of all of the user's channels? If I include a channels.scm file in a project, it would be nice if it only contained the channels necessary to build and run the packages within that project. Is it possible to use the guix.scm or manifest.scm file when generating the channels.scm file so it only includes the necessary channels? Would I need to create a temporary project profile first and then use that to filter the channel list? ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Reproducible Project Environments 2021-12-08 14:30 ` Peter Polidoro @ 2021-12-09 20:33 ` Ludovic Courtès 0 siblings, 0 replies; 8+ messages in thread From: Ludovic Courtès @ 2021-12-09 20:33 UTC (permalink / raw) To: Peter Polidoro; +Cc: zimoun, guix-science Hi Peter, Peter Polidoro <peter@polidoro.io> skribis: >> >> From my point of view, if we are speaking of a project gluing many >> thing >> to produce analysis, then a Git repo with: >> >> - manifest.scm containing package set for the project >> - channels.scm containing the revision >> - all your files > > Does 'guix describe - f channels' always return a list of all of the > user's channels? If I include a channels.scm file in a project, it > would be nice if it only contained the channels necessary to build and > run the packages within that project. Yes, it includes all the channels your ‘guix’ command is using, but you could easily edit it to remove unnecessary channels. > Is it possible to use the guix.scm or manifest.scm file when > generating the channels.scm file so it only includes the necessary > channels? Would I need to create a temporary project profile first and > then use that to filter the channel list? I think the workflow in general would be: 1. Write ‘manifest.scm’ (or ‘guix.scm’). 2. Run ‘guix describe -f channels > channels.scm’. 3. If you know that ‘channels.scm’ contains more than what’s strictly needed, remove it. In theory it should be possible to automatically determine from the manifest which channels are definitely required, but there currently isn’t a command to do the filtering in step #3 for yourself. HTH! Ludo’. ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2021-12-09 20:33 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-12-06 14:18 Reproducible Project Environments Peter Polidoro 2021-12-06 16:16 ` Julien Lepiller 2021-12-06 18:09 ` Peter Polidoro 2021-12-06 19:52 ` Efraim Flashner 2021-12-06 16:58 ` zimoun 2021-12-06 18:13 ` Peter Polidoro 2021-12-08 14:30 ` Peter Polidoro 2021-12-09 20:33 ` Ludovic Courtès
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).