From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1gomvr-0005fS-2Y
	for mharc-gwl-devel@gnu.org; Wed, 30 Jan 2019 05:17:23 -0500
Received: from eggs.gnu.org ([209.51.188.92]:53273)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <zimon.toutoune@gmail.com>) id 1gomvo-0005bw-TX
	for gwl-devel@gnu.org; Wed, 30 Jan 2019 05:17:21 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <zimon.toutoune@gmail.com>) id 1gomvj-0006AE-JO
	for gwl-devel@gnu.org; Wed, 30 Jan 2019 05:17:17 -0500
Received: from mail-qk1-x732.google.com ([2607:f8b0:4864:20::732]:46266)
	by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16)
	(Exim 4.71) (envelope-from <zimon.toutoune@gmail.com>)
	id 1gomvj-00069F-El
	for gwl-devel@gnu.org; Wed, 30 Jan 2019 05:17:15 -0500
Received: by mail-qk1-x732.google.com with SMTP id q1so13315959qkf.13
	for <gwl-devel@gnu.org>; Wed, 30 Jan 2019 02:17:14 -0800 (PST)
MIME-Version: 1.0
References: <87bm40qta0.fsf@elephly.net> <875zu7refm.fsf@elephly.net>
	<CAJ3okZ2TU+mWt72CYOJ9cMA6kR1muXDTG86ejyQ5Tyo5DQvHGg@mail.gmail.com>
	<87womnptym.fsf@elephly.net>
	<CAJ3okZ03DTudrdpoD30VqvuWqK0iS8uUBjRPqisDrpGVQLvRiA@mail.gmail.com>
	<874l9rpeiq.fsf@elephly.net>
	<CAJ3okZ1dhN4uD2sSEYjSbhHnNwkc9c+_g_X=H+2y6Xn08aZ9eA@mail.gmail.com>
	<87womnnjg0.fsf@elephly.net>
In-Reply-To: <87womnnjg0.fsf@elephly.net>
From: zimoun <zimon.toutoune@gmail.com>
Date: Wed, 30 Jan 2019 11:17:02 +0100
Message-ID: <CAJ3okZ0QGwo6dzdYwEpMBauYN2vYgFMuYMR2jAuk=nTGoLhkZg@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Subject: Re: support for containers
List-Id: <gwl-devel.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/gwl-devel>,
	<mailto:gwl-devel-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/gwl-devel/>
List-Post: <mailto:gwl-devel@gnu.org>
List-Help: <mailto:gwl-devel-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/gwl-devel>,
	<mailto:gwl-devel-request@gnu.org?subject=subscribe>
To: Ricardo Wurmus <rekado@elephly.net>
Cc: gwl-devel@gnu.org

Hi Ricardo,

On Wed, 30 Jan 2019 at 00:16, Ricardo Wurmus <rekado@elephly.net> wrote:

> Since we don=E2=80=99t hash the data (because it=E2=80=99s expensive) the=
 scripts are
> =E2=80=9Cproxies=E2=80=9D for the data files.  We compute the hashes over=
 the dependent
> scripts and assume that this is enough to decide whether to recompute
> data files or to serve them from the cache/store.

Just to be sure to well understand your point, let pick the simple
example from genomics pipeline:
 FASTQ -align-> BAM -variant-> VCF
So, you intend to hash:
 - the data FASTQ
 - the scripts align and variant
Or only the scripts containing reference to inputs (here FASTQ), where
the reference is a location fixed by the user.

Well, hashing the scripts and assuming they "mirror" the data files
appear to me an efficient design for the CAS.

--
simon