From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:8:6d80::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id IagUDc3YZmDPHAEAgWs5BA (envelope-from ) for ; Fri, 02 Apr 2021 10:41:49 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id EJjeBc3YZmAVUgAAB5/wlQ (envelope-from ) for ; Fri, 02 Apr 2021 08:41:49 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 0B79D12DD6 for ; Fri, 2 Apr 2021 10:41:46 +0200 (CEST) Received: from localhost ([::1]:35654 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lSFNA-0004xW-IX for larch@yhetil.org; Fri, 02 Apr 2021 04:41:44 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44036) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lSFN8-0004xB-3B for gwl-devel@gnu.org; Fri, 02 Apr 2021 04:41:42 -0400 Received: from wout2-smtp.messagingengine.com ([64.147.123.25]:45561) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lSFN6-0002eR-4a for gwl-devel@gnu.org; Fri, 02 Apr 2021 04:41:41 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id 77E9CD47; Fri, 2 Apr 2021 04:41:38 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Fri, 02 Apr 2021 04:41:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastmail.net; h= from:to:cc:subject:in-reply-to:references:date:message-id :mime-version:content-type:content-transfer-encoding; s=fm3; bh= +yyse+C6KEB0FJ5WRowCHjxeBb3yS7x87efC+6T8L2I=; b=O2LO/dZMyvnZLjwp mQ82jbevNsvf1IfKMjCMNplaXX7UTSTkipt3+gJ82SNe7oUUmlSg6bVEgT2z8P4e GgIQtXoVxle8ByUytNAb6dxlPSzGhm8vyMqckAhVSO3Pz+OTre9E5FAuCi03VdPk DjHladD1tJKiVh3xuvu8H4nPks8T57W1G1i2xbN8mlzT2MFMFa2dTIA5HsOZRGS3 6u41z4jIgMeXjZk29rhaXbwmMZvEIcxSQJpKZh+zFLj3a/0k6itoDf8XjnjJV54e mT0Pm/kCKRkTzeSViHX0u7Ut0EFW3MY87YxnIsO6seS/xrAslVizoIvEOmMR0w8U PEt8MA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=+yyse+C6KEB0FJ5WRowCHjxeBb3yS7x87efC+6T8L 2I=; b=O9VQ5yt0qSaoZrApzJdyy5oAh2K2yDKzp9MgYJxxVbxtkgVNNxnFGo53X JtunCISWU1XI94NFf7fY4vUjJZ5dui/gAwFbUVLTGYstoD5s9RO5eHt/NElHWaC/ G+f2Ky5T2syVynyBf7ZfyUWiLhV//PR9P48hPpZcjugFW7jpj0JP64BUpVuL+f05 7VC35xo3puq2J6yjwQlsvoV7WFpPJGaYRb3POHeFUyyVENrydXqgqQesOP7c406P /kSmWvXpPA6JlZkcvyBcG3Fbf0PLs2kAnt06NOTx8BW1xqofrVA52Tjcl8JB8f30 4WCM/irrZFl+kECBWCZqBvaPDy3sA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudeiiedgtdeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffujghffffkgggtgfesthhqre dttddtjeenucfhrhhomhepmfhonhhrrgguucfjihhnshgvnhcuoehkohhnrhgrugdrhhhi nhhsvghnsehfrghsthhmrghilhdrnhgvtheqnecuggftrfgrthhtvghrnhepveehleevge ejuddvgfduieefveetffdvteegfeekgfduueejfeegvdehgeekhfejnecukfhppeekiedr vdegjedrgeekrdehkeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrih hlfhhrohhmpehkohhnrhgrugdrhhhinhhsvghnsehfrghsthhmrghilhdrnhgvth X-ME-Proxy: Received: from ordinateur-de-catherine--konrad.home (lfbn-idf2-1-840-58.w86-247.abo.wanadoo.fr [86.247.48.58]) by mail.messagingengine.com (Postfix) with ESMTPA id 001E61080066; Fri, 2 Apr 2021 04:41:36 -0400 (EDT) From: Konrad Hinsen To: Ricardo Wurmus Subject: Re: Managing data files in workflows In-Reply-To: <87h7kq2kzy.fsf@elephly.net> References: <87r1k2ti7k.fsf@elephly.net> <87czvmt5w3.fsf@elephly.net> <87h7kq2kzy.fsf@elephly.net> Date: Fri, 02 Apr 2021 10:41:35 +0200 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=64.147.123.25; envelope-from=konrad.hinsen@fastmail.net; helo=wout2-smtp.messagingengine.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: gwl-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: gwl-devel@gnu.org Errors-To: gwl-devel-bounces+larch=yhetil.org@gnu.org Sender: "gwl-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1617352908; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=+yyse+C6KEB0FJ5WRowCHjxeBb3yS7x87efC+6T8L2I=; b=QPuUjw2GyStL01TzhvoWRk7xlTmpDsBCjzZ2OjOTbGm1fJtWGOvWDAfqVfkWH9BHczS9+3 7x2HtGKBYsr8Wi7oFJsQF3Aj1JKTbKDYXfwGz2Oqhb5cV4n4z+8uTy7HXFo2kj4PEP6wtX EOyNvGcTQ46amgHGds7H2QyG4VUdPe6u+XtLLXRQleyI7Fbkp7MQ4P9suiV5xQHS5MdEPk v2GnJMO0oqnPKz/zRFppoojeOXkVegVDbmd9qqYx5J4Wb2Wag9z53bB8DijI5jXG4B9LwY aGkTTtryEkP8Tc/W+59HV/LdpmHEzcoqiYFaSqLHsdQlRzWAUzMwRHIzkzYbHg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1617352908; a=rsa-sha256; cv=none; b=sAJf+WjvWMqG4udbMou9n2hvLSIKYgmq/Cs8VFYXM0dIU3mKKQFw/elg8BXbiTFXdDjeuQ xVxlgFYGxiT0LZ7CW6CqSQpceHPSmDjIrq1NNKOAwhTJifOfuTOAWn9BI8s+HF2w9uzZ7i 5Jz9LbCQcIvdT6+6HJb/QeV37esBzQvtSk20/dLOhr8c3gsiQ0VLB7veLMwqw8PDyaknpv MnqTA1IgibGs/AUE8vlJV02houa2C62KFqM+W/FU/zoig/ywDc/gdSDIoOF7Ma8e82Dgl9 G6RxKySiYNJpGYxxWz/ZOmdEmJAyOfOLnddV1zywa74bSlM51kDy3O/jfoU++w== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=fastmail.net header.s=fm3 header.b="O2LO/dZM"; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=O9VQ5yt0; dmarc=pass (policy=none) header.from=fastmail.net; spf=pass (aspmx1.migadu.com: domain of gwl-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=gwl-devel-bounces@gnu.org X-Migadu-Spam-Score: -3.13 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=fastmail.net header.s=fm3 header.b="O2LO/dZM"; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=O9VQ5yt0; dmarc=pass (policy=none) header.from=fastmail.net; spf=pass (aspmx1.migadu.com: domain of gwl-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=gwl-devel-bounces@gnu.org X-Migadu-Queue-Id: 0B79D12DD6 X-Spam-Score: -3.13 X-Migadu-Scanner: scn0.migadu.com X-TUID: 4b2+nRGvsvEH Hi Ricardo, > Maybe. You could run with =E2=80=9C--dry-run=E2=80=9D to see what GWL cl= aims it would > do to confirm that it considers the file to be =E2=80=9Cnot cached=E2=80= =9D. > > Also enable more log events (in particular cache events) with > > =E2=80=9C--log-events=3Derror,info,execute,cache,debug=E2=80=9D Thanks, I think I made progress with those nice debugging aids. When I run my workflow for the first time, I see cache: Caching `./data/weekly-incidence.csv' as `/tmp/gwl/mwmeuuhnu7sv4mpouj7o5x4se4qp5n5auzhpkb7y7oxidoxzc6ra/./data/wee= kly-incidence.csv' The '.' in there looks suspect. Let's see what I got: $ ls -lR /tmp/gwl/mwmeuuhnu7sv4mpouj7o5x4se4qp5n5auzhpkb7y7oxidoxzc6ra /tmp/gwl/mwmeuuhnu7sv4mpouj7o5x4se4qp5n5auzhpkb7y7oxidoxzc6ra: total 4 drwxrwxr-x 2 hinsen hinsen 4096 2 avril 10:13 data /tmp/gwl/mwmeuuhnu7sv4mpouj7o5x4se4qp5n5auzhpkb7y7oxidoxzc6ra/data: total 0 lrwxrwxrwx 1 hinsen hinsen 27 2 avril 10:13 weekly-incidence.csv -> ./d= ata/weekly-incidence.csv That's an invalid symbolic link, so it's not surprising that a second run doesn't find the cached file. When I use an absolute filename to refer to my download target, the symlink in the cache is valid and points to the downloaded file. And when I run the workflow a second time, it skips the "download" process as expected. But then, it fails trying to "restore" the file: run: Skipping process "download" (cached at /tmp/gwl/ubvscxwoezl63qmvyfs= zlf6azmuc655h7gbbtosqshlm5r6ckyhq/). cache: Restoring `/tmp/gwl/ubvscxwoezl63qmvyfszlf6azmuc655h7gbbtosqshlm5= r6ckyhq//home/hinsen/projects/mooc-workflows/influenza-analysis/data/weekly= -incidence.csv' to `/home/hinsen/projects/mooc-workflows/influenza-analysis= /data/weekly-incidence.csv' Backtrace: 6 (primitive-load "/home/hinsen/.config/guix/current/bin/guix= ") In guix/ui.scm: 2164:12 5 (run-guix-command _ . _) In srfi/srfi-1.scm: 460:18 4 (fold # =E2=80=A6) 460:18 3 (fold # =E2=80=A6) In gwl/workflows.scm: 392:21 2 (_ # ()) In srfi/srfi-1.scm: 634:9 1 (for-each # =E2=80=A6) In guix/ui.scm: 566:4 0 (_ system-error "symlink" _ _ _) guix/ui.scm:566:4: In procedure symlink: Operation not permitted: "/home= /hinsen/projects/mooc-workflows/influenza-analysis/data/weekly-incidence.cs= v" Looking at the source code in (gwl cache), restoring means symlinking the target file to the cached file, which can't work given that the cache is already a symlink to the target file. So... I don't understand how the cache is supposed to work. If it stores symlinks, there is no need to restore anything. If it is supposed to store copies, then that's not what it does. My original issue with the relative filename is a detail that should be easy to fix. Cheers, Konrad.