From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id GB0fLqRCrmGHYAEAgWs5BA (envelope-from ) for ; Mon, 06 Dec 2021 18:04:36 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id uNmyKaRCrmEyawAAbx9fmQ (envelope-from ) for ; Mon, 06 Dec 2021 17:04:36 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 219DC7AB2 for ; Mon, 6 Dec 2021 18:04:34 +0100 (CET) Received: from localhost ([::1]:35382 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1muHPl-0002lg-Pm for larch@yhetil.org; Mon, 06 Dec 2021 12:04:33 -0500 Received: from eggs.gnu.org ([209.51.188.92]:38672) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1muHPg-0002gQ-54 for guix-science@gnu.org; Mon, 06 Dec 2021 12:04:28 -0500 Received: from [2a00:1450:4864:20::330] (port=55903 helo=mail-wm1-x330.google.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1muHPd-00062K-FO for guix-science@gnu.org; Mon, 06 Dec 2021 12:04:27 -0500 Received: by mail-wm1-x330.google.com with SMTP id p18so8673776wmq.5 for ; Mon, 06 Dec 2021 09:04:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:subject:in-reply-to:references:date:message-id:mime-version :content-transfer-encoding; bh=eHhLgi7cCZa/Y6Qe+aaDZaIDoErDOBGUJIpJzzRoj/w=; b=VJQNfV9ZkIAb8LxlUzNnW1pE2VAnDeZ/wyOxVm0U1NjM1TyNy5Vf/DhdBhctrvUklc YIWJhiaRgcgVkCCAbN4p04mPJew/H3TEM8qfFCEWt0YDUS9PrG5HfFkaBZuh8Wu0ICUo HsJIEddgHE3O/SBvUrw6bdH2fBYM5CFxoeKLP1bpVdGWNEN3T/9/koXBsXyfSn+Njeh4 HWLtk0irBayYs1J1U+lscsARFXcx5JwU+opJIDxOnzGmY/eV2gxjJskqELCLyh7qngrS JlF+SVW2odtAGHdtugvyqadTHmJt/Uwxv8P+Fg++I29UhCXtnNWJIjtRVUtdV8SDID3r 7d9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:in-reply-to:references:date :message-id:mime-version:content-transfer-encoding; bh=eHhLgi7cCZa/Y6Qe+aaDZaIDoErDOBGUJIpJzzRoj/w=; b=R4R1UHB5yslnPcEBFadPFUq8mVbo19+oLE2vXRhkJqknOjM8wHfiezOXp3/zpowOjG RPkXKtkxHkA8rXeAfJ5uaRR/yC0+FpkJvfDBy7tcdgndZnqqH5lBgTKQGhbYos1nBcQp 3YjoxlBCDMF5ffkm87VIs0RsE1lzp/2JjAJ9qWj2XlL+dQ4dMTe2MkD0InKfKUWEmrMm YSmNB343AAUlN6PKtkGz+Wd8iUd0JonnjV0i3N/38jnKnFIt1O/O7u75JyAj31BZQpT9 a8vicHRyAOQz5GB3KsxJf4KdWWNu4yiqVxPpVAbLLKTS5sBliiwqT6jvaFTS21nbcTDl 5MmA== X-Gm-Message-State: AOAM531sCXeH6m0NFOMyDG6KhD7x9iqXcStAM+kUpszzp/WgbPBgYdz2 QVEKHmWCGZ/XUqarmHvfKSw4Mh2LG3A= X-Google-Smtp-Source: ABdhPJzeJuQ8/+z7g1vyXVZlEYMMF4QGqzznHp5z6g2zhmtyw4WVUHZ4AO3jv+UdqM6P7W7L+AIkEA== X-Received: by 2002:a7b:c194:: with SMTP id y20mr41619367wmi.61.1638810263278; Mon, 06 Dec 2021 09:04:23 -0800 (PST) Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e]) by smtp.gmail.com with ESMTPSA id b15sm14799844wri.62.2021.12.06.09.04.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Dec 2021 09:04:22 -0800 (PST) From: zimoun To: Peter Polidoro , guix-science@gnu.org Subject: Re: Reproducible Project Environments In-Reply-To: <86lf0x3f9c.fsf@polidoro.io> References: <86lf0x3f9c.fsf@polidoro.io> Date: Mon, 06 Dec 2021 17:58:09 +0100 Message-ID: <86fsr5oela.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Host-Lookup-Failed: Reverse DNS lookup failed for 2a00:1450:4864:20::330 (failed) Received-SPF: pass client-ip=2a00:1450:4864:20::330; envelope-from=zimon.toutoune@gmail.com; helo=mail-wm1-x330.google.com X-Spam_score_int: -12 X-Spam_score: -1.3 X-Spam_bar: - X-Spam_report: (-1.3 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, PDS_HP_HELO_NORDNS=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-science@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-science-bounces+larch=yhetil.org@gnu.org Sender: "Guix-Science" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1638810275; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=eHhLgi7cCZa/Y6Qe+aaDZaIDoErDOBGUJIpJzzRoj/w=; b=agrX3iBpDy0PL4bm+c2usTaA8yO7WG0p6AJpXWyjA3kvBcL8mvx8XkEuH8bcMyOfiX8ieB NrlwDq0bDY6/8twuLIcZBTG1f907VwxZy55hIzHQ/FbTLx037iPnpSOlN0O/kCuexrCNP0 77U/8UBB/3pjRNCOGB68ZxJ9mqFHgnHhyNMMrTTiDi7hNNOQd8bMpTIqOCh7ByrGpUdutb oeXfkNU9ut4qgycuKVFrWHxdE/vgF1ExUTMHvNGQOwYQdqNbhdHc/Jxc6IdqNMILBJfQTp aNHb67DNB+C0w/VvBG7KKbNNNEQymrfkAee0i+3Ykw3Pl2aL8i9QbjpuNJY8Ng== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1638810275; a=rsa-sha256; cv=none; b=Ki3YwfIlW3kCQmkYkUstpadKs/KWjwvcyKDAG5oZTuB+PJkFB9rRS2pAX20YF/n43aERzn BRfuzONUTjzeJpnRC5/2gl1eyzG5pdORzQ43HaIyLlxN1QZkc20TfO5e9IUQLVaZuBZwaH rqulpXBlTu8GgOCd0QU42WFGf69DzdYuhEdlv+v91h7QGkJwOoGXIvR3lLL6qHg3W97T8P gz3Z8F/m6QjUUAqYvXXjjkqpGcmCB/ohpNsY7kNcW3ZOPgwwH3xkKCRZHDHCeldD4y5RXG Y6qVOmvSlgUlDcHNUmaw+IU28h4GPYY66Hm39yeg/ZucGdqf5yr0EZmXtOkM9w== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=VJQNfV9Z; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-science-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-science-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -4.14 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=VJQNfV9Z; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "guix-science-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-science-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 219DC7AB2 X-Spam-Score: -4.14 X-Migadu-Scanner: scn0.migadu.com X-TUID: ZyA4LwsKGJUS Hi, On Mon, 06 Dec 2021 at 09:18, Peter Polidoro wrote: > Guix is a wonderful way to distribute software, especially for=20 > science. Thank you all for such a fantastic project and community! > > I am an engineer working with scientists at the Howard Hughes=20 > Medical Institute, a science philanthropy based in the United=20 > States funding researchers all over the world. I hope to help=20 > convince the researchers I work with to all use Guix someday. Cool! > My question is, what files do you recommend placing within a=20 > project directory to make it as straightforward as possible to=20 > distribute that project to others to develop and use in a=20 > reproducible environment? >From my point of view, if we are speaking of a project gluing many thing to produce analysis, then a Git repo with: - manifest.scm containing package set for the project - channels.scm containing the revision - all your files If some package are not yet in Guix, or cannot be, they should be in a custom channel (another Git repo). All Git repos must be saved in Software Heritage. Note that channels.scm describes all the channels (official Guix and all custom channels) and each state of the channel are fixed by commit. guix time-machine -C channels.scm -- shell -m manifest.scm will do a lot of job. :-) If a Git repo disappears for some reasons, then the command would fallback to SWH, more or less. ;-) > What do you recommend when some of the project dependencies are=20 > not yet Guix packages? Obviously it would be best to get those=20 > dependencies into Guix as official packages eventually, but what=20 > is the best way to handle those dependencies until then? IMHO, custom channels. https://hpc.guix.info/channels/ > Say that in addition to some Guix packages, the project also needs=20 > some Python packages that are available on pypi, but are not yet=20 > Guix packages. Using something like direnv, an .envrc file in the=20 > project directory can install the Guix package dependencies into a=20 > local profile, then pip install the Python dependencies into a=20 > project local virtualenv. Then the user just needs Guix and direnv=20 > installed on their machine to get a reproducible environment when=20 > they change into that directory. Specifying dependencies in an=20 > .envrc file does not seem as elegant as using a pure Guix=20 > solution, however. >From a reproducibility point of view, this method is =E2=80=9Cdangerous=E2= =80=9D but there is no guarantee that PyPI would be in the same state. On a side note, there is no guarantee either for transparency. For instance, https://hpc.guix.info/blog/2021/09/whats-in-a-package/ > What is the best way to handle these dependencies without using=20 > direnv and just Guix? Should I create Guix packages for the Python=20 > dependencies in an external custom channel? If so, how do you=20 > specify custom channels when using guix shell? Is there a method=20 > that is as simple as direnv or does it involve a user separately=20 > setting up a new profile to reference the custom channel before=20 > running guix shell? Yes, I would create a custom channel for these dependencies. Starting with =E2=80=9Cguix import pypi -r=E2=80=9D. Yeah, it can be a lot of work;= if not impossible for some stuff found in PyPI. Well, it depends on your level of transparency and the complexity of your stack. As I wrote above, guix time-machine -C channels.m -- shell -m manifest.scm appears to me the good practise. Note that manifest.scm could contain transformation or quickly define a variant using inherit. > Do you ever create packages within a guix.scm file or a=20 > manifest.scm file or perhaps in a separate channel.scm file in the=20 > project directory to keep the project more self contained than=20 > referencing an external custom channel? Then would you create some=20 > project local profile that references this file to use when=20 > running guix shell? Or is the recommended approach to always use a=20 > custom channel before submitting the packages to be included into=20 > Guix proper? Thanks! To include packages to Guix proper, the package must comply with the standards. https://guix.gnu.org/manual/en/guix.html#Packaging-Guidelines If for some reasons, the package cannot be included to Guix proper, it could be included to one of these channels, https://hpc.guix.info/channels/ which avoid too many package scattering. ;-) And bonus, they often provide substitutes. Thanks for your message. Cheers, simon