From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id oPlJF/f+j2DCEQAAgWs5BA (envelope-from ) for ; Mon, 03 May 2021 15:47:35 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id UFgIE/f+j2BdNQAA1q6Kng (envelope-from ) for ; Mon, 03 May 2021 13:47:35 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id E205F1A8F4 for ; Mon, 3 May 2021 15:47:32 +0200 (CEST) Received: from localhost ([::1]:46284 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ldYv5-0002Oh-QL for larch@yhetil.org; Mon, 03 May 2021 09:47:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47626) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ldYv4-0002O6-9F for gwl-devel@gnu.org; Mon, 03 May 2021 09:47:30 -0400 Received: from sender4-of-o51.zoho.com ([136.143.188.51]:21133) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ldYv2-00039l-Im for gwl-devel@gnu.org; Mon, 03 May 2021 09:47:30 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1620049646; cv=none; d=zohomail.com; s=zohoarc; b=IRRwtkfugGcQwQ5VLX0F1LY5rFxX6AxQ5Z18/t17ACPxrTbIELVD/SrZjMyYYtqpL1ALZkPDGp6ZVL8UqiBAR9rVRYC80B245qeNS8/qehx67C9UVnchcH8lxhTzsczHoW7qSwDmTYR7H97hzldDB684yMDbD5WDSAjgf9zZ09U= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1620049646; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=ZuMHj3KCb5BrH3lmPLUZWSEurpF6C6AMiRAnjMyhT5M=; b=Hc78q/WyGUFqzMmWohz4zl+IP+dlMX1uTfXAb9JPrEfPxb9+M8pnyGuuRGXeJYSKSdsbW+tLVcEqmg056Ii9XTV/wdWFGRdCB1HhMQfohE49TUsynRtk1nxvO+jcJNLL/P5qNBjqV5QoUbWvYHTb64MuWVZSU27ZLRlayqI0pfU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1620049646; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; bh=ZuMHj3KCb5BrH3lmPLUZWSEurpF6C6AMiRAnjMyhT5M=; b=ARCVTyLQYne11yoERybgaolhhnZe9+xnQVQbYoBXsDn5a+2kXVrnuT4+YB+TZoqQ RCjF/LjAUoQnEYMP/yQNKj6K18ukrZczFI1cTotDyqEusPLN8ML+1j93D1qoVj/QstS rpXhhpNxOQMHAhhEE5t5zPuhv6ZNMzFeZcQSarrc= Received: from localhost (p4fd5aec5.dip0.t-ipconnect.de [79.213.174.197]) by mx.zohomail.com with SMTPS id 1620049645066710.6075981506937; Mon, 3 May 2021 06:47:25 -0700 (PDT) References: <87r1k2ti7k.fsf@elephly.net> <87czvmt5w3.fsf@elephly.net> <87h7kq2kzy.fsf@elephly.net> <87blaqz5mq.fsf@elephly.net> <87a6pckwf9.fsf@elephly.net> <86eeeovxjj.fsf@gmail.com> User-agent: mu4e 1.4.15; emacs 27.2 From: Ricardo Wurmus To: zimoun Subject: Re: Managing data files in workflows In-reply-to: <86eeeovxjj.fsf@gmail.com> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Mon, 03 May 2021 15:47:22 +0200 Message-ID: <87y2cwj5ed.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External Received-SPF: pass client-ip=136.143.188.51; envelope-from=rekado@elephly.net; helo=sender4-of-o51.zoho.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: gwl-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: gwl-devel@gnu.org Errors-To: gwl-devel-bounces+larch=yhetil.org@gnu.org Sender: "gwl-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1620049653; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=ZuMHj3KCb5BrH3lmPLUZWSEurpF6C6AMiRAnjMyhT5M=; b=QsFxgD4F4sbsOMWCtyXWwOz5HKXZ2DA/+YdNJhsCnmjq0PGNR/VPcEhUcYq/573xUn1ziB 0Ki53hBoZOSLFMnmOODRft6U/6+ytIy7bCqMZxTYYTUlxhk1A/NYkRXFlXp2RC++5l0kAX agOMGdef1Uwc15ueGPswYR5zromg8W/MJnkjpkS6FQ+92WIQqqFBl6wMCLuaPprmI4U/Wz t+Lp3fxdPDysIgkYqsL45a2/t66dxnlask6j8SXFJ86R4vZqlKMDOgezvvyOsREOBGYTSx kYNGsg3VDgA5gcGh9EYdAhii0mR61vZKG8Yf32iPIRePRPP4QOpOQ0RpLgVn+g== ARC-Seal: i=2; s=key1; d=yhetil.org; t=1620049653; a=rsa-sha256; cv=fail; b=l9B2OCGPrgmihcnJgl3xOaHymUKJkB8NtIaA5UrEiIuDxJv3srtk9upycWpx3QUFJXA75H RPaZyPCDdJ8wQ5AHlthVnVaNslxxsayJfIhwojEdsyhywDUAkbjCxREXpJySFashv/FzQD IBD7z4Hw7R5hnG+iMSRVWVl1jBoyOcmAPOUzUl3QJffGjv2RWEsVvWRp0l1nQcTuiBuLZW z4aP0YrhP4jlVw9WPlTqgvxv4y02e1p3fqVYv/xR+q1G4UEbjdVOKf4nzn91Fr8FeADNYD CdzuugAK8ZA/C3IyaWQP/N8AhLxfMw0vAXKy91QPgZRLCW+jdWjHHTEJmWMy8w== ARC-Authentication-Results: i=2; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=elephly.net header.s=zoho header.b=ARCVTyLQ; arc=reject ("signature check failed: fail, {[1] = sig:zohomail.com:reject}"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of gwl-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=gwl-devel-bounces@gnu.org X-Migadu-Spam-Score: 0.54 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=elephly.net header.s=zoho header.b=ARCVTyLQ; arc=reject ("signature check failed: fail, {[1] = sig:zohomail.com:reject}"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of gwl-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=gwl-devel-bounces@gnu.org X-Migadu-Queue-Id: E205F1A8F4 X-Spam-Score: 0.54 X-Migadu-Scanner: scn0.migadu.com X-TUID: V/gqdF0ZTxj1 zimoun writes: > Hi, > > On Mon, 03 May 2021 at 11:18, Ricardo Wurmus=20 > wrote: > >> I=E2=80=99d like to introduce modifiers =E2=80=9Cimmutable=E2=80=9D and = =E2=80=9Cmutable=E2=80=9D, so=20 >> that=20 >> you can write =E2=80=9Cimmutable file "whatever" you "want"=E2=80=9D etc= .=20 >> =E2=80=9Cimmutable=E2=80=9D would take care of recording hashes and chec= king=20 >> previously recorded hashes in a local state directory. > > Ah, maybe it is same idea that we discussed some days ago on=20 > #guix-hpc. > > To me, everything should be =E2=80=9Cimmutable=E2=80=9C and stored =E2=80= =9Csomewhere=E2=80=9D. > Somehow, you do not need to hash all the contents but only hash=20 > all the > inputs hashes and the =E2=80=99process=E2=80=99 itself. This is in fact what we do now. Every process code snippet is=20 hashed, and we compute the hashes of the chain of processes to=20 determine whether to re-run something or not. > However, I miss what =E2=80=9Cmutable=E2=80=9D could be? It=E2=80=99s a very generous assumption that intermediate output files are= =20 reproducible. Often that=E2=80=99s not actually the case. Having a=20 =E2=80=9Cmutable=E2=80=9D modifier would allow us to indicate that this is = in fact=20 the case and require recomputation. --=20 Ricardo