Re: Managing data files in workflows

[Ricardo Wurmus <rekado@elephly.net>]

zimoun <zimon.toutoune@gmail.com> writes:

> Hi,
>
> On Mon, 03 May 2021 at 11:18, Ricardo Wurmus 
> <rekado@elephly.net> wrote:
>
>> I’d like to introduce modifiers “immutable” and “mutable”, so 
>> that 
>> you can write “immutable file "whatever" you "want"” etc. 
>> “immutable” would take care of recording hashes and checking 
>> previously recorded hashes in a local state directory.
>
> Ah, maybe it is same idea that we discussed some days ago on 
> #guix-hpc.
>
> To me, everything should be “immutable“ and stored “somewhere”.
> Somehow, you do not need to hash all the contents but only hash 
> all the
> inputs hashes and the ’process’ itself.

This is in fact what we do now.  Every process code snippet is 
hashed, and we compute the hashes of the chain of processes to 
determine whether to re-run something or not.

> However, I miss what “mutable” could be?

It’s a very generous assumption that intermediate output files are 
reproducible.  Often that’s not actually the case.  Having a 
“mutable” modifier would allow us to indicate that this is in fact 
the case and require recomputation.

-- 
Ricardo