Reproducible Project Environments

Reproducible Project Environments

[Peter Polidoro]

Guix is a wonderful way to distribute software, especially for 
science. Thank you all for such a fantastic project and community!

I am an engineer working with scientists at the Howard Hughes 
Medical Institute, a science philanthropy based in the United 
States funding researchers all over the world. I hope to help 
convince the researchers I work with to all use Guix someday.

My question is, what files do you recommend placing within a 
project directory to make it as straightforward as possible to 
distribute that project to others to develop and use in a 
reproducible environment?

If all of the project dependencies are in Guix as packages, then a 
guix.scm or manifest.scm file seems all that is needed to create a 
reproducible environment using guix shell, as long as someone has 
Guix installed on their machine.

What do you recommend when some of the project dependencies are 
not yet Guix packages? Obviously it would be best to get those 
dependencies into Guix as official packages eventually, but what 
is the best way to handle those dependencies until then?

Say that in addition to some Guix packages, the project also needs 
some Python packages that are available on pypi, but are not yet 
Guix packages. Using something like direnv, an .envrc file in the 
project directory can install the Guix package dependencies into a 
local profile, then pip install the Python dependencies into a 
project local virtualenv. Then the user just needs Guix and direnv 
installed on their machine to get a reproducible environment when 
they change into that directory. Specifying dependencies in an 
.envrc file does not seem as elegant as using a pure Guix 
solution, however.

What is the best way to handle these dependencies without using 
direnv and just Guix? Should I create Guix packages for the Python 
dependencies in an external custom channel? If so, how do you 
specify custom channels when using guix shell? Is there a method 
that is as simple as direnv or does it involve a user separately 
setting up a new profile to reference the custom channel before 
running guix shell?

Do you ever create packages within a guix.scm file or a 
manifest.scm file or perhaps in a separate channel.scm file in the 
project directory to keep the project more self contained than 
referencing an external custom channel? Then would you create some 
project local profile that references this file to use when 
running guix shell? Or is the recommended approach to always use a 
custom channel before submitting the packages to be included into 
Guix proper? Thanks!

Re: Reproducible Project Environments

[Julien Lepiller]

[-- Attachment #1: Type: text/plain, Size: 365 bytes --]

When my projects are missing a dependency from guix, or use a specific version, I simply specify them in my guix.scm for tge project, so it's just as easy to create the environment. I remove it as soon as I contribute it to guix. I would only use a separate channel if multiple projects require the same dependency and it.s taking a lot of time to get it into guix.

[-- Attachment #2: Type: text/html, Size: 406 bytes --]

Re: Reproducible Project Environments

[Peter Polidoro]

Julien Lepiller <julien@lepiller.eu> writes:

> When my projects are missing a dependency from guix, or use a 
> specific version, I simply specify them in my guix.scm for tge 
> project, so it's just as easy to create the environment.

Do you have any examples you can share of using both Guix official 
packages and custom package specifications in a guix.scm file? 
Thanks!

Re: Reproducible Project Environments

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1033 bytes --]

On Mon, Dec 06, 2021 at 01:09:23PM -0500, Peter Polidoro wrote:
> 
> Julien Lepiller <julien@lepiller.eu> writes:
> 
> > When my projects are missing a dependency from guix, or use a specific
> > version, I simply specify them in my guix.scm for tge project, so it's
> > just as easy to create the environment.
> 
> Do you have any examples you can share of using both Guix official packages
> and custom package specifications in a guix.scm file? Thanks!

The most complex ones I know of are for gash-utils¹ and one I wrote for
fennel². As far as a simpler one, I use this one³ for my guix.vim
plugin.

¹ https://git.savannah.gnu.org/cgit/gash/gash-utils.git/tree/guix.scm
² https://git.sr.ht/~efraim/fennel/tree/main/item/guix.scm
³ https://git.sr.ht/~efraim/guix.vim/tree/master/item/guix.scm

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Reproducible Project Environments

[zimoun]

Hi,

On Mon, 06 Dec 2021 at 09:18, Peter Polidoro <peter@polidoro.io> wrote:
> Guix is a wonderful way to distribute software, especially for 
> science. Thank you all for such a fantastic project and community!
>
> I am an engineer working with scientists at the Howard Hughes 
> Medical Institute, a science philanthropy based in the United 
> States funding researchers all over the world. I hope to help 
> convince the researchers I work with to all use Guix someday.

Cool!


> My question is, what files do you recommend placing within a 
> project directory to make it as straightforward as possible to 
> distribute that project to others to develop and use in a 
> reproducible environment?

From my point of view, if we are speaking of a project gluing many thing
to produce analysis, then a Git repo with:

  - manifest.scm containing package set for the project
  - channels.scm containing the revision
  - all your files

If some package are not yet in Guix, or cannot be, they should be in a
custom channel (another Git repo).

All Git repos must be saved in Software Heritage.

Note that channels.scm describes all the channels (official Guix and all
custom channels) and each state of the channel are fixed by commit.

   guix time-machine -C channels.scm -- shell -m manifest.scm

will do a lot of job. :-)  If a Git repo disappears for some reasons,
then the command would fallback to SWH, more or less. ;-)


> What do you recommend when some of the project dependencies are 
> not yet Guix packages? Obviously it would be best to get those 
> dependencies into Guix as official packages eventually, but what 
> is the best way to handle those dependencies until then?

IMHO, custom channels.

    https://hpc.guix.info/channels/


> Say that in addition to some Guix packages, the project also needs 
> some Python packages that are available on pypi, but are not yet 
> Guix packages. Using something like direnv, an .envrc file in the 
> project directory can install the Guix package dependencies into a 
> local profile, then pip install the Python dependencies into a 
> project local virtualenv. Then the user just needs Guix and direnv 
> installed on their machine to get a reproducible environment when 
> they change into that directory. Specifying dependencies in an 
> .envrc file does not seem as elegant as using a pure Guix 
> solution, however.

From a reproducibility point of view, this method is “dangerous” but
there is no guarantee that PyPI would be in the same state.

On a side note, there is no guarantee either for transparency.  For
instance,

    https://hpc.guix.info/blog/2021/09/whats-in-a-package/


> What is the best way to handle these dependencies without using 
> direnv and just Guix? Should I create Guix packages for the Python 
> dependencies in an external custom channel? If so, how do you 
> specify custom channels when using guix shell? Is there a method 
> that is as simple as direnv or does it involve a user separately 
> setting up a new profile to reference the custom channel before 
> running guix shell?

Yes, I would create a custom channel for these dependencies.  Starting
with “guix import pypi -r”.  Yeah, it can be a lot of work; if not
impossible for some stuff found in PyPI.  Well, it depends on your level
of transparency and the complexity of your stack.

As I wrote above,

    guix time-machine -C channels.m -- shell -m manifest.scm

appears to me the good practise.  Note that manifest.scm could contain
transformation or quickly define a variant using inherit.


> Do you ever create packages within a guix.scm file or a 
> manifest.scm file or perhaps in a separate channel.scm file in the 
> project directory to keep the project more self contained than 
> referencing an external custom channel? Then would you create some 
> project local profile that references this file to use when 
> running guix shell? Or is the recommended approach to always use a 
> custom channel before submitting the packages to be included into 
> Guix proper? Thanks!

To include packages to Guix proper, the package must comply with the
standards.

    https://guix.gnu.org/manual/en/guix.html#Packaging-Guidelines

If for some reasons, the package cannot be included to Guix proper, it
could be included to one of these channels,

    https://hpc.guix.info/channels/

which avoid too many package scattering. ;-)  And bonus, they often
provide substitutes.


Thanks for your message.


Cheers,
simon

Re: Reproducible Project Environments

[Peter Polidoro]

zimoun <zimon.toutoune@gmail.com> writes:

>
>    guix time-machine -C channels.scm -- shell -m manifest.scm
>

This seems like a nice clean way of doing it once you have a 
custom channel set up.

>
> If for some reasons, the package cannot be included to Guix 
> proper, it
> could be included to one of these channels,
>
>     https://hpc.guix.info/channels/
>

Perhaps I will create a custom channel for our organization and 
then try to move packages from there into these other channels or 
into Guix proper once they have been tested.

Thanks you!

Re: Reproducible Project Environments

[Peter Polidoro]

>
> From my point of view, if we are speaking of a project gluing 
> many thing
> to produce analysis, then a Git repo with:
>
>   - manifest.scm containing package set for the project
>   - channels.scm containing the revision
>   - all your files

Does 'guix describe - f channels' always return a list of all of 
the user's channels? If I include a channels.scm file in a 
project, it would be nice if it only contained the channels 
necessary to build and run the packages within that project.

Is it possible to use the guix.scm or manifest.scm file when 
generating the channels.scm file so it only includes the necessary 
channels? Would I need to create a temporary project profile first 
and then use that to filter the channel list?

Re: Reproducible Project Environments

[Ludovic Courtès]

Hi Peter,

Peter Polidoro <peter@polidoro.io> skribis:

>>
>> From my point of view, if we are speaking of a project gluing many
>> thing
>> to produce analysis, then a Git repo with:
>>
>>   - manifest.scm containing package set for the project
>>   - channels.scm containing the revision
>>   - all your files
>
> Does 'guix describe - f channels' always return a list of all of the
> user's channels? If I include a channels.scm file in a project, it
> would be nice if it only contained the channels necessary to build and
> run the packages within that project.

Yes, it includes all the channels your ‘guix’ command is using, but you
could easily edit it to remove unnecessary channels.

> Is it possible to use the guix.scm or manifest.scm file when
> generating the channels.scm file so it only includes the necessary 
> channels? Would I need to create a temporary project profile first and
> then use that to filter the channel list?

I think the workflow in general would be:

  1. Write ‘manifest.scm’ (or ‘guix.scm’).

  2. Run ‘guix describe -f channels > channels.scm’.

  3. If you know that ‘channels.scm’ contains more than what’s strictly
     needed, remove it.

In theory it should be possible to automatically determine from the
manifest which channels are definitely required, but there currently
isn’t a command to do the filtering in step #3 for yourself.

HTH!

Ludo’.