From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:8:6d80::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id wmsDId2ZbWD0ugAAgWs5BA (envelope-from ) for ; Wed, 07 Apr 2021 13:39:09 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id vcX4Gd2ZbWA4OAAA1q6Kng (envelope-from ) for ; Wed, 07 Apr 2021 11:39:09 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 8FE031C20C for ; Wed, 7 Apr 2021 13:39:08 +0200 (CEST) Received: from localhost ([::1]:55136 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lU6WZ-0000Jg-9k for larch@yhetil.org; Wed, 07 Apr 2021 07:39:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:40836) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lU6WL-00007w-Fm for gwl-devel@gnu.org; Wed, 07 Apr 2021 07:38:53 -0400 Received: from sender4-of-o51.zoho.com ([136.143.188.51]:21156) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lU6WH-0006mL-62 for gwl-devel@gnu.org; Wed, 07 Apr 2021 07:38:53 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1617795522; cv=none; d=zohomail.com; s=zohoarc; b=a5SC0CG6afZT8cP9p3I5HKIsWOs96rCl4lrGEG+vpzsCV90dRp7jrRA8rbNlqvwRKbnbnZKV7iX6Hcd6RDaoL5Wy4CVhXs0EGyuTbbbtjRoBuGgHrl78H6GmCiqKtLLQDixe2pTi9XWfu6r11bbfJdHMfqFiMTRfMxTYdisqmIY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1617795522; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=dASDVouJEaE4N86Ysd54C8vvvM0MDgf/lYMKY8FT+Wk=; b=WcxoOmd2cRSan7fCJMCSNLbxJ9TWZ5awh0y16XaKiZdwwtrhNcVzihotbef/cnFJqRTV1b6PxhhyWCdTzXNx1PAFTgE7xQha4XW1kvr3dJ3s5RxeiXtt47dunkjsuksmikOKOU2VtQP+KIrUuQDsL72RIhe4aW9wBGsGaPCey8I= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1617795522; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; bh=dASDVouJEaE4N86Ysd54C8vvvM0MDgf/lYMKY8FT+Wk=; b=CTFS6AdBBHFHbyu+72EcqapOjdjYyf2BJR1JNGSXNCtugjklh9JdGeflA2BhNdI5 cGsK0RCTseffTxO4xDN+kf03k/fx/pGeRMtdsRJY5z89Upl+TdCbMv7xDPN/xZcQAis IhNZVCMzIOr2vjJiXmXj/N5xr0eRfHtOp21yLCqw= Received: from localhost (p54ad4b0c.dip0.t-ipconnect.de [84.173.75.12]) by mx.zohomail.com with SMTPS id 1617795520958267.6791455168342; Wed, 7 Apr 2021 04:38:40 -0700 (PDT) References: <87r1k2ti7k.fsf@elephly.net> <87czvmt5w3.fsf@elephly.net> <87h7kq2kzy.fsf@elephly.net> User-agent: mu4e 1.4.15; emacs 27.2 From: Ricardo Wurmus To: Konrad Hinsen Subject: Re: Managing data files in workflows In-reply-to: X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Wed, 07 Apr 2021 13:38:37 +0200 Message-ID: <87blaqz5mq.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External Received-SPF: pass client-ip=136.143.188.51; envelope-from=rekado@elephly.net; helo=sender4-of-o51.zoho.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: gwl-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: gwl-devel@gnu.org Errors-To: gwl-devel-bounces+larch=yhetil.org@gnu.org Sender: "gwl-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1617795549; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=dASDVouJEaE4N86Ysd54C8vvvM0MDgf/lYMKY8FT+Wk=; b=rQNcBwkRQSZCZ/ZcksGJreYrMN7yopIWrqxDnHpx4QC/2EyWaPuFSgGx/92TVcH5RvdUtf vT70WzMjeBciARYnG+bRAOZrBEyug0XG1FkSUB8sv/4JvOoh68kwUiPTfdB54RNDWz00s5 jqQJToT+MPRelUjn8Zcqzld/6X8EO5vNmzGyaD5EhBRtxfMXBOzccnxQ41ozs9rRpAsDm/ KTWxo7cVzFKrw4G+56k6yKpXe+rn93//ytssj4MhMz4Z2q7Q9gl8y9fDA8ICInwSYyVsxl NFgtM1ggfpIVTov8goxsbKKd6cYk30E7nAdN6+90cDxWAQ4JjEsQTF/ghsx0rg== ARC-Seal: i=2; s=key1; d=yhetil.org; t=1617795549; a=rsa-sha256; cv=pass; b=OuekQN0oWopHwnXX2ic12Eb04YeGxzZm3K2qFSQR/VokHDYZna5YqvM3RzEy04lgoPxTYy MHEMStLqRWEPxf0WdihJNngOrQMghPzSOdZpIBPGHrFOQHKoUu63EW5cu8i1iD9PS30Mr+ 7nSjNYSpmHqREST+ZGZb55Fb8qpFnSs31ARZDr9Xrn2nU7r3B3oC/FZjf3r4VW+6+xy0kB EaW6Cwap/LWgETiVgCTRYPV2IXLZJV0EWrIRojJB/2y5JE+ZTnU1ugdzu6Qd2Nxoy4ZGl8 phBH7UoQsbFuOO1zpJdsQbyEVk8elBqXBms+OmpTtckRd79Loj01cJj78g1A8w== ARC-Authentication-Results: i=2; aspmx1.migadu.com; dkim=pass header.d=elephly.net header.s=zoho header.b=CTFS6AdB; arc=pass ("zohomail.com:s=zohoarc:i=1"); spf=pass (aspmx1.migadu.com: domain of gwl-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=gwl-devel-bounces@gnu.org X-Migadu-Spam-Score: -3.64 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=elephly.net header.s=zoho header.b=CTFS6AdB; arc=pass ("zohomail.com:s=zohoarc:i=1"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of gwl-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=gwl-devel-bounces@gnu.org X-Migadu-Queue-Id: 8FE031C20C X-Spam-Score: -3.64 X-Migadu-Scanner: scn0.migadu.com X-TUID: IoqeaBOFhPXX Konrad Hinsen writes: > Looking at the source code in (gwl cache), restoring means symlinking > the target file to the cached file, which can't work given that the > cache is already a symlink to the target file. > > So... I don't understand how the cache is supposed to work. If it stores > symlinks, there is no need to restore anything. If it is supposed to > store copies, then that's not what it does. Right, that=E2=80=99s really the heart of the problem here. Originally, I = used hardlinks exclusively, but they don=E2=80=99t work everywhere. So I added symlinks, but obviously they have different semantics. We can fix the problem with symlinks by restoring the target of the link instead of the link itself, but I feel that we need to take a step back and consider what this cache is really to be used for. The cache assumes again that files are immutable when really they are not guaranteed to be immutable. Both symlinks and hardlinks don=E2=80=99t = give us any guarantees. I really would like to have independent copies of input and output files, but I also don=E2=80=99t want to needlessly copy files around or use= up more space than absolutely necessary. We could punt on the problem of optimal space consumption and simply copy files to the cache. What do you think? --=20 Ricardo