From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id cBlTAJ5tGWQHMQEASxT56A (envelope-from ) for ; Tue, 21 Mar 2023 09:41:02 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id MNUFO51tGWTTKwEAG6o9tA (envelope-from ) for ; Tue, 21 Mar 2023 09:41:01 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id DBDCB276A for ; Tue, 21 Mar 2023 09:41:00 +0100 (CET) Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=fastmail.net header.s=fm2 header.b=N19bxmAa; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm2 header.b=l3FCG3zQ; dmarc=fail reason="SPF not aligned (relaxed)" header.from=fastmail.net (policy=none); spf=pass (aspmx1.migadu.com: domain of "gwl-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="gwl-devel-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1679388061; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=ieFoYqL8CqlHtJNkWt6UCHsJJLF45f1FTU5Yo85idCQ=; b=SH0tNDGW4LrV49OtmClFcA0b0H7Rnh8WHlGlUrsr3iyL1NfyfkNgD/K0dZ0p9ubU0aBkcV h9YZOySXbIpTsOFQfSPKY6GnRYWfJPHf2FTn5uK/RxMTt0RstDvhCF4lHmA3UxS07hFAGN mC7CVGMr3ZlYc6igYdOzZZ9WZ/PnVdPmQ+EYdoVsHH9hy605VCpkQQGCMZymkYqIibBdGQ 7nweVwUb6rAT2Bhlr1e9fvv4w/00VvU7rLTHKAZFpF5WwWYEcRebHqjgrEMoLRh9q8yfiJ f2FYPpIGcyl9bMYZmCoUgCr8vQxWwBdCCaqXA9YgofnZUFkcb7eBoxu6uScZdg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=fastmail.net header.s=fm2 header.b=N19bxmAa; dkim=fail ("headers rsa verify failed") header.d=messagingengine.com header.s=fm2 header.b=l3FCG3zQ; dmarc=fail reason="SPF not aligned (relaxed)" header.from=fastmail.net (policy=none); spf=pass (aspmx1.migadu.com: domain of "gwl-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="gwl-devel-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1679388061; a=rsa-sha256; cv=none; b=YSZPodWhxVIxiq6jTSFinje+e+ZGRhEUw272Xud3ktGEWUs1N5FqIDZqHA8wF7pNmdaQW2 JAnY4DEWRnFhFP4cJXYRJjOqnMrnRKg15nnc2wj6P8wfsSDNVSai+T7w0Own4aKRrn+gGw OqpcOzY1ig1qj+pLqK9R+m1RVBArz5jiE5+el34a/AEDXjdQUsrNhLta4EPlsr18w4soFG XKqc33ZXxp8aZzG/ixsERM9mMXLawHykhnWhnuPNl76jjWVWBZdzrdvgMz9p5YqiVGKX2+ iZsjJHmwJx6BNP8CQt9eP2vZbTGV8mDGaBaRU91jUeYSdLxEssHEJGkqWIhEmQ== Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1peX7z-00020G-3j; Tue, 21 Mar 2023 04:13:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1peX7x-0001zs-8A for gwl-devel@gnu.org; Tue, 21 Mar 2023 04:13:53 -0400 Received: from wout4-smtp.messagingengine.com ([64.147.123.20]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1peX7v-0003C4-72 for gwl-devel@gnu.org; Tue, 21 Mar 2023 04:13:53 -0400 Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id EA5FE3200645; Tue, 21 Mar 2023 04:13:45 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Tue, 21 Mar 2023 04:13:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastmail.net; h= cc:cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm2; t= 1679386425; x=1679472825; bh=ieFoYqL8CqlHtJNkWt6UCHsJJLF45f1FTU5 Yo85idCQ=; b=N19bxmAaSUGQE4h5VjxOQ6Enbt0uOLEhDbVeJv+0FvUyzAGrRiN ZUEdtRYcow6yKhOM35/2y32KOn4qztH+y5JR7F/nyL1Yos+WhOArs7Vyyj0CnSuS EvevIWVEexNikxOW8U+fGfMyJ7CH4A0AgDH8OH0DtMPpxD6lTri4HcZnjOGTFj+w /s69wbDUSLPf1RYIdYSAbYyuv4UgPZPNzIAIpKt1D2bBBQtxpKAt4gblZWdKPNZs 0tMaC1kANRTjNrs49UPWXTYH4xPnK31YMiX4+BHB5pGYeDif819esWNDeMAGJQAt B1u72nvIYzlUuF+77VcSWPo34Dr5J81fQxg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1679386425; x=1679472825; bh=ieFoYqL8CqlHtJNkWt6UCHsJJLF45f1FTU5 Yo85idCQ=; b=l3FCG3zQpO9w97SSaqTueYGuZjGAbBqtuplvKBCEhS0JUyyQqDc UgPQqDNrtNgBMPWnSE6mF0x+vQ71mR3U5Ou8on0rDktL0SX+kfmDaZaOZiWV2+4F FvltsumMC9eEecRzA1iZ8xzERjzKIa/X5uXWk/pQiOSxNwBSxCsYTJZgCCVVKA2S OGqkTZP0M29G+EUasO8MuTC7j4m2ipxbXXwSAtUbOEh1QE53yVa8/9K1B5fZta/m ppswMR8kH/7/y9v8dqGQ91/FVZaC0VsXsw9TtXF8NORFQHuj2GhkmQ6Fg/BXQQmX 2YgFtvrytubXxJprYWdhySHHRjzDKQdFu+g== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdefledguddugecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhephffvvefujghffffkgggtgfesth hqredttddtjeenucfhrhhomhepmfhonhhrrgguucfjihhnshgvnhcuoehkohhnrhgrugdr hhhinhhsvghnsehfrghsthhmrghilhdrnhgvtheqnecuggftrfgrthhtvghrnhepieehvd elhedujeejvdefudehieeifeegueeiieegteehffduleelgfegueeifeelnecuvehluhhs thgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepkhhonhhrrggurdhhih hnshgvnhesfhgrshhtmhgrihhlrdhnvght X-ME-Proxy: Feedback-ID: i184641e2:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 21 Mar 2023 04:13:44 -0400 (EDT) From: Konrad Hinsen To: Simon Tournier , gwl-devel@gnu.org Cc: Ricardo Wurmus Subject: Re: Containerized workflow in containerized processes In-Reply-To: <87wn3klsrr.fsf@gmail.com> References: <87wn3klsrr.fsf@gmail.com> Date: Tue, 21 Mar 2023 09:13:43 +0100 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=64.147.123.20; envelope-from=konrad.hinsen@fastmail.net; helo=wout4-smtp.messagingengine.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: gwl-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: X-Migadu-Queue-Id: DBDCB276A X-Spam-Score: -2.95 X-Migadu-Spam-Score: -2.95 X-Migadu-Scanner: scn0.migadu.com List-Help: List-Subscribe: , Errors-To: gwl-devel-bounces+larch=yhetil.org@gnu.org Sender: gwl-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-TUID: xdrGhSxQOUcN Hi Simon, > The use-case =E2=80=9CContainerized workflow in containerized processes= =E2=80=9D appears > to me interesting. :-) > > It is almost done by design with GWL, no? That's an interesting observation. But I don't see anything in the GWL manual that explains how GWL manages processes.=20 The chapter "Process engines" says: "The simplest way is to turn the workflow into a Guile script that sets up the desired environment and then executes the workflow processes on the current machine. Fine, but is that script run in a container? If not, the programs and code snippets from that process could run arbitrary binaries from the file system. In the examples shown, the workflow itself is launched from the command line, so it is not running in a container either. In principle, the Guile script defining the workflow could access arbitrary files, and thus not be reproducible. I suspect that the risk is low in practice, because I see no good reason for doing this. Cheers, Konrad > > -------------------- Start of forwarded message -------------------- > From: Konrad Hinsen > To: Simon Tournier , Guix Devel > Subject: Re: Using Guix inside a Guix container > Date: Sat, 18 Feb 2023 10:21:52 +0100 > > Hi Simon, > >> Which part of Guix do you need inside the containerized shell that you >> cannot do outside? > > That's not the right question. There's always a way to do what I want to > do outside. But that may be very inconvenient. > >> Considering your use-case with Snakemake, what I am doing is to wrap >> each rule with one containerized Guix shell which controls the >> permissions, rule by rule; or a big containerized shell: >> >> guix shell -C -m manifest.scm --expose=3D=E2=80=A6 > > Nice example. I do the same: "guix shell" in every rule. Then I add > stuff to my Snakefile, which is a Python script after all. For example, > I import pandas to read a data frame from which I construct my workflow. > Now I am at the point where I'd like to run snakemake itself in a > container, to manage the dependencies of my Snakefile. In fact, given > that I have workflows that depend on specific Snakemake versions, I'd > really like to run Snakemake in a container all the time, even without > additional dependencies. > > Without nested containers, I have to go through all the rules, collect > the packages from their manifest files (or command line), and add them > to the container in which I run the whole workflow. Possible, but not > convenient. > > Another example: I run command-line programs from my Pharo image, and I > have developed the habit of doing this always through Guix. The > advantage is that my Pharo code becomes portable: it depends on Guix, > but not on my profile. > > But if I want, one day, to move on to a full Guix system, I have to run > Pharo in a container with LFS simulation. And then all my command line > shell-outs will break. > > Both examples are about composing tools freely, without worrying if they > use Guix internally or now. > > Cheers, > Konrad > > -------------------- End of forwarded message -------------------- > > Cheers, > simon