From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:bcc0::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id 2AubEv/dXWAQ6wAAgWs5BA (envelope-from ) for ; Fri, 26 Mar 2021 14:13:35 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id 8HWDDP/dXWCkTgAAB5/wlQ (envelope-from ) for ; Fri, 26 Mar 2021 13:13:35 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 6A64A27433 for ; Fri, 26 Mar 2021 14:13:34 +0100 (CET) Received: from localhost ([::1]:43926 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lPmHM-00030s-3m for larch@yhetil.org; Fri, 26 Mar 2021 09:13:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36102) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lPmHJ-0002zK-Er for gwl-devel@gnu.org; Fri, 26 Mar 2021 09:13:29 -0400 Received: from sender4-of-o51.zoho.com ([136.143.188.51]:21192) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lPmHH-0000yQ-Ji for gwl-devel@gnu.org; Fri, 26 Mar 2021 09:13:29 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1616764403; cv=none; d=zohomail.com; s=zohoarc; b=DSa14A5jg5pkx4Q60K9HJTajkwmXufcYHmhNQd3LfW9IqPPGBe5F4iXPJoxfNFglfnPIDAn029GRu6o0Z5gXdvX/5PRM0218DQwtsx+XNGiK0ab3ElD+te339z6sfnCJLP5Pl32ZzbcM/6cq5IEylN4YGJVrfs8NF4MiafmqZKk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1616764403; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=uVqVsFEvAhP3RDMD7gqIt/wpfm/NnMHsqdaCO2iUJe8=; b=R9A6JpzTYAbt3pRmFEyap/7o7fUbzzQ3aIdsOrsyd4XpG5GpEc1AHWjM3lY9jNi6u3Ij05BxBv0IDitdxdKFM8I+i3h5CR/IA9MKNZLC+TISjHchapwtktYjBzvmUYNP1SoaCE4z416xQQwb+2MtStc41If/G18PedvrEpyVviY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1616764403; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; bh=uVqVsFEvAhP3RDMD7gqIt/wpfm/NnMHsqdaCO2iUJe8=; b=CGIZKNiDgXE6I0I2aWu9NL/V1FNNQfYsOlbIcISy7ipZuJWCQ1nsIqGkTLQpN0pe mn6tciHJ1e4lAbaKYyrH2fEJCGJ8KSlsEXq9tEXVeFtlejOm5XDnfoX8FgCrHguxVqJ TpxTfnKUVSQ4XAu3vDmP8kQ1tTFJlsTvybG4UgJk= Received: from localhost (p54ad4990.dip0.t-ipconnect.de [84.173.73.144]) by mx.zohomail.com with SMTPS id 1616764399516656.6997212688043; Fri, 26 Mar 2021 06:13:19 -0700 (PDT) References: <87r1k2ti7k.fsf@elephly.net> User-agent: mu4e 1.4.14; emacs 27.1 From: Ricardo Wurmus To: Konrad Hinsen Subject: Re: Managing data files in workflows In-reply-to: X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Fri, 26 Mar 2021 14:13:16 +0100 Message-ID: <87czvmt5w3.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External Received-SPF: pass client-ip=136.143.188.51; envelope-from=rekado@elephly.net; helo=sender4-of-o51.zoho.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: gwl-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: gwl-devel@gnu.org Errors-To: gwl-devel-bounces+larch=yhetil.org@gnu.org Sender: "gwl-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1616764415; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=uVqVsFEvAhP3RDMD7gqIt/wpfm/NnMHsqdaCO2iUJe8=; b=rzam55hNQDp1RA1KvAadvbxZqEutKbRn2kv1PUpxAti281A5qWTdcrxVNnx1ktt90/bl0n xfBQgqzxwt95PMb8f7Pgb2ALr6uGE2GySbI0sbcuof3pS3Ud+M6+GB8Iogy4u9bUOsVYKh cHEin7eSIjAYMgeCDEm9+ZVVJXSRWSzfSZ2pLhL+Awfy6xG38yLpxsTCaGWxyyztusSM30 gEeqBIcUJn3R6Vb+XbQgXOkaI9b2WuLI6vF9gKBpfE118lX9C+lm30u8RinPxKVheUAhie DY5mYfe7nhMXgwru5zDMJ8baxKWuPMQBWyWTR6LSNbxchEkwyYyo8f8ySKuV4g== ARC-Seal: i=2; s=key1; d=yhetil.org; t=1616764415; a=rsa-sha256; cv=pass; b=Y2ZDl0ju2loldT1dgZLMpStPeBWzfUa7LC2LPh1020iXL9UHfhGovCRc0X82YbWVWRJm4q TGFPvUCTMJ9iFauPGXKSr+witJgad1NFepWe+6wEzJ3+HSGlltF5CTiNAsmHCyt75b3GR0 r2EypeCDtmjYu0RiX/HiK7IPslwAwbcmm3pl9ACmPzM1+ABStUYt6iZYK7Fa657iTii2hZ 98xWN8CdwFKdbGcVWkYYPpJZTdF4cna1nbGLmUSENkhuLCGcDMD/lPOwlSyQvkgHQKN9Qh f1HZWtfnwN/3QcAewOf++lU+mPIqFqe7zMs6BGP8tJ63JgNie634mx9QUiGFSg== ARC-Authentication-Results: i=2; aspmx1.migadu.com; dkim=pass header.d=elephly.net header.s=zoho header.b=CGIZKNiD; arc=pass ("zohomail.com:s=zohoarc:i=1"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of gwl-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=gwl-devel-bounces@gnu.org X-Migadu-Spam-Score: -4.17 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=elephly.net header.s=zoho header.b=CGIZKNiD; arc=pass ("zohomail.com:s=zohoarc:i=1"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of gwl-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=gwl-devel-bounces@gnu.org X-Migadu-Queue-Id: 6A64A27433 X-Spam-Score: -4.17 X-Migadu-Scanner: scn0.migadu.com X-TUID: rZqY1UOydLOE Konrad Hinsen writes: >> This works for me correctly: > > Thanks for looking into this! For me, your change makes no difference. > Nor should it, because in my setup the "data" directory already exists. > I still get an error message about the already existing file. > > Maybe it's time to switch to the development version of GWL! Hmm, I don=E2=80=99t see any commits since 0.3.0 that would affect the cache implementation. GWL computes cache hashes for all processes and the processes they depend on. In your case it=E2=80=99s trivial: there=E2=80= =99s just one process. The process definition is hashed and looked up in the cache to see if there is any output for the given process hash. In my test case this file exists: /tmp/gwl/lf6uca7zcyyldkcrxn3zwc275ax3ip676aqgjo75ybwojtl4emoq/data/week= ly-incidence.csv /tmp/gwl is the cache prefix, and the hash corresponds to the process. Since data/weekly-incidence.csv exists and that=E2=80=99s the only declared output, GWL decides not compute the output again. At least that happens in my case. I wonder why it doesn=E2=80=99t work in = your case. > However, what I had in mind with my question is the management of > intermediate results in my workflow, especially in its development > phase. If I change my workflow file, or a script that it calls, > I'd want only the affected steps to be recomputed. That's not much > of an issue for my current test case, but I have bigger dreams for > the future ;-) Yes, that=E2=80=99s the way it=E2=80=99s supposed to work already. GWL com= putes the hashes of each chain of processes, which includes the generated process script, its inputs, and the hashes of all processes that lead up to this process. Any change in the chain will lead to a new hash and thus a cache miss, leading GWL to recompute. --=20 Ricardo