From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from list by lists.gnu.org with archive (Exim 4.71) id 1goQo0-0006Lt-Ef for mharc-gwl-devel@gnu.org; Tue, 29 Jan 2019 05:39:48 -0500 Received: from eggs.gnu.org ([209.51.188.92]:55529) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1goQnx-0006La-Ug for gwl-devel@gnu.org; Tue, 29 Jan 2019 05:39:46 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1goQnu-0007zV-4R for gwl-devel@gnu.org; Tue, 29 Jan 2019 05:39:45 -0500 Received: from mail-qk1-x742.google.com ([2607:f8b0:4864:20::742]:43519) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1goQnq-0007w7-Am for gwl-devel@gnu.org; Tue, 29 Jan 2019 05:39:40 -0500 Received: by mail-qk1-x742.google.com with SMTP id z18so11169626qkj.10 for ; Tue, 29 Jan 2019 02:39:36 -0800 (PST) MIME-Version: 1.0 References: <87bm40qta0.fsf@elephly.net> <875zu7refm.fsf@elephly.net> In-Reply-To: <875zu7refm.fsf@elephly.net> From: zimoun Date: Tue, 29 Jan 2019 11:39:24 +0100 Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: support for containers List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Ricardo Wurmus Cc: gwl-devel@gnu.org Hi Ricardo, On Tue, 29 Jan 2019 at 10:54, Ricardo Wurmus wrote: > Ricardo Wurmus writes: > > > the GWL could already support execution in containers with this patch: > [=E2=80=A6] > > The directories to be mounted in the container are derived from the > > declared inputs and outputs. The only problem is that inputs are > > read-only in this implementation. I like it this way, actually, but it > > means that the extended example workflow won=E2=80=99t work as it tries= to > > delete its inputs. > > > > Should data inputs be declared as (mutable-file =E2=80=A6) or (file =E2= =80=A6) instead > > of being plain strings? > > Taking a step back I think it is worth remembering that ultimately we=E2= =80=99d > like to have output caching via an immutable data store. I think > declaring files as mutable would be a mistake. Garbage collection is > better than modifying output files. I agree that Garbage Collection is a better design, from my opinion. However, as an user I would like to garbage collect only the inputs/outputs of a specific workflow. Not all inside the store. > Some thoughts on how the data store should work: it=E2=80=99s easy to add= stuff > to the store (we just hash the inputs leading up to the output, > excluding the output itself); but how do we make store items available > to a process? When using containers we can bind mount the file to the > declared input location; when not using containers we need to link or > copy the file from By store, do you the Guix store? or another store? as GWL store? By inputs, do you mean data-inputs and package-inputs? I do not know if below makes sense. What happens if a kind of record managing the data-inputs is added? I mean, there are: package, process, workflow and we add data. Then the data is manage as other symbols, with define-public, etc. However, these data need to live a special store, not in the store of packa= ges. > > I also want to have finer control over inputs. Only declared input > *files* should be available in the container, not whole *directories*. Hum? I make sense. Especially for references used by aligners. Even if the size of the container will significantly growth. :-) All the best, simon