From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id twoNHnb/XWD/NQEAgWs5BA (envelope-from ) for ; Fri, 26 Mar 2021 16:36:22 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id gEvcFnb/XWDmFwAAbx9fmQ (envelope-from ) for ; Fri, 26 Mar 2021 15:36:22 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id C2B659C93 for ; Fri, 26 Mar 2021 16:36:20 +0100 (CET) Received: from localhost ([::1]:34042 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lPoVX-0006A7-Te for larch@yhetil.org; Fri, 26 Mar 2021 11:36:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46018) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lPoVV-00068f-GY for gwl-devel@gnu.org; Fri, 26 Mar 2021 11:36:17 -0400 Received: from out5-smtp.messagingengine.com ([66.111.4.29]:60061) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lPoVP-00020b-H5 for gwl-devel@gnu.org; Fri, 26 Mar 2021 11:36:17 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id CAFBA5C00D3; Fri, 26 Mar 2021 11:36:10 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Fri, 26 Mar 2021 11:36:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastmail.net; h= from:to:cc:subject:in-reply-to:references:date:message-id :mime-version:content-type:content-transfer-encoding; s=fm3; bh= NReDy2ITxBiXNsk6v+1mOLPx40rBd2FUtCanh3fNWSE=; b=iONwjFFCMp5idJ+O ShO1Nzi3GS+CS2f53iHLVkvrdHHQKUG6m2JQimtIxgXPU8b4PklFver5QCXmbu1+ cq88EeNG8w0/5flviCh4rThkFrozUZicmvtzQcY4fvKZnBBOTbUpuxOFyC+6jcpm JM2cEwieu4r40UPPzO2SnrIJvnKb/TJVQVeu3MF5KFGNOa9yt372BspNTPxima3o /ksJKAzCqYhLO9KrM7/e/VD04B+GE8UicV5CiC1Ogb51KXmC+AkuOmRrydOVsLEU lXx9c4NQj9ZaBnWJz8SrdTIRzhRxQrCFs4v86ODl+oL7F0KsAcyXXnt21f1WXT0V 5XwYEw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=NReDy2ITxBiXNsk6v+1mOLPx40rBd2FUtCanh3fNW SE=; b=vbqRe1qsa2OJoCzhNZDeR66T44DN3zA/AsxG2aNNyay2SXZt2FO4qOgEW u7IY584g0+4j4h53W3cRh5MvRDT7Orv8Udj/lzSlq7tl4TI53GZ34Gjv3ku8BByp 4ywoTF1Bx6igExW5y2vcPFSZ2u8VqyIqIUpOxmtET6+cGtP7Plmcagr53ZZeQ+7m 0MmkhjzWeTAg3rCXEnOa49aaTdVuwXjVO1p3pK+RiGuq/rnU+WIGSWZFubZ4mQO3 z5kPM+mtVZegZP/lVLhdJ+ptjvJFWNKGd8PaJ9CI6IllFWaA7rqezlohAwQX/nno 0IXWAZZ4xotlqTRRRL4rWe7Zk+65w== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudehvddgkedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffujghffffkgggtgfesthhqre dttddtjeenucfhrhhomhepmfhonhhrrgguucfjihhnshgvnhcuoehkohhnrhgrugdrhhhi nhhsvghnsehfrghsthhmrghilhdrnhgvtheqnecuggftrfgrthhtvghrnheptdfgheetke egkeetjeeutedvvdehvddvleelfeffteehhfevteevudekfedtkeeknecuffhomhgrihhn pehsvghnthhifigvsgdrfhhrnecukfhppeekiedrvdegjedrgeekrdehkeenucevlhhush htvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehkohhnrhgrugdrhhhi nhhsvghnsehfrghsthhmrghilhdrnhgvth X-ME-Proxy: Received: from ordinateur-de-catherine--konrad.home (lfbn-idf2-1-840-58.w86-247.abo.wanadoo.fr [86.247.48.58]) by mail.messagingengine.com (Postfix) with ESMTPA id 3DF0B1080057; Fri, 26 Mar 2021 11:36:09 -0400 (EDT) From: Konrad Hinsen To: Ricardo Wurmus Subject: Re: Managing data files in workflows In-Reply-To: <87czvmt5w3.fsf@elephly.net> References: <87r1k2ti7k.fsf@elephly.net> <87czvmt5w3.fsf@elephly.net> Date: Fri, 26 Mar 2021 16:36:08 +0100 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=66.111.4.29; envelope-from=konrad.hinsen@fastmail.net; helo=out5-smtp.messagingengine.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: gwl-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: gwl-devel@gnu.org Errors-To: gwl-devel-bounces+larch=yhetil.org@gnu.org Sender: "gwl-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1616772982; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=NReDy2ITxBiXNsk6v+1mOLPx40rBd2FUtCanh3fNWSE=; b=TVxOBldAAtSIv9eIrKvluBl2zkFtK+vH4Uwap10BuQpomc7rb4lNiaJpLjAM2jIIsXfo+d s1ABZ/mOzpHoHlCDlVMH9vCPv4nnt5CLvNX04m3u1noNZSL86Mhd1s6g8ELycDZqaSbqXp 6LD31fRxWkUP9eElKlza2ZkFilJ+UMsKCscHFYqovzI/nMozptcPZ4ZhEvSxdniLLmHlvW pOqeECTV1t6hwkHnC+s8mhC44TzJs2G054NkHBDxzK35NHrr/qKje8SQBGFyBJHP1sB711 xJZ/loT4kgt6kaAUDd6awYMafTp/TglxUxhcmjab525Bc5jIxPzDou8GFKBp+w== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1616772982; a=rsa-sha256; cv=none; b=SPMW8IAoy5qJbmEugeP8GLpqqhywJDsxO+vn/yCkDt1EAzNWSFhOLGjsAuYd2l/npTHTBP q5aCyLGeZf7HTkbMKJVLruYL/IVz/H4kbFepLtZ6jYqUHDexGxvooaRkwOZC78mwyeEYw1 Qg6kKPjGWLz7bQKCh2xss+9aKWNgfUyMbztpQAJv97hLsgjkyfr2kHcnPDslR/wI710USy nHTYaOAyksnlkz5q0cIrPqjO8Uaxzhtg5YJr53d2IwSPYwUa+K9giuyhijD9HKYKKgeR3k H0TVByBoJjRlPunbBGVvIdAqMh6lDytkrxSttUrv3SSZiJvjVq9ucX+efuNuqw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=fastmail.net header.s=fm3 header.b=iONwjFFC; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=vbqRe1qs; spf=pass (aspmx1.migadu.com: domain of gwl-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=gwl-devel-bounces@gnu.org X-Migadu-Spam-Score: -3.12 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=fastmail.net header.s=fm3 header.b=iONwjFFC; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=vbqRe1qs; dmarc=pass (policy=none) header.from=fastmail.net; spf=pass (aspmx1.migadu.com: domain of gwl-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=gwl-devel-bounces@gnu.org X-Migadu-Queue-Id: C2B659C93 X-Spam-Score: -3.12 X-Migadu-Scanner: scn0.migadu.com X-TUID: QoYHuZtnRWy1 Ricardo Wurmus writes: > In my test case this file exists: > > /tmp/gwl/lf6uca7zcyyldkcrxn3zwc275ax3ip676aqgjo75ybwojtl4emoq/data/we= ekly-incidence.csv And that's the one that causes the error message for me when I run the workflow a second time (see below). But as I understand now, the mistake happens earlier, as this step shouldn't be executed at all. > At least that happens in my case. I wonder why it doesn=E2=80=99t work i= n your > case. Is there anything I can do to debug this? > Yes, that=E2=80=99s the way it=E2=80=99s supposed to work already. GWL c= omputes the > hashes of each chain of processes, which includes the generated process > script, its inputs, and the hashes of all processes that lead up to this > process. Any change in the chain will lead to a new hash and thus a > cache miss, leading GWL to recompute. Excellent, that's what I was hoping to happen, given the Guix background. Cheers, Konrad. $ guix workflow run test.w info: Loading workflow file `test.w'... info: Computing workflow `influenza-incidence'... run: Executing: /bin/sh -c /gnu/store/km977swwhqj2n1mg34fq6sv4a41iabkm-gwl-= download.scm '((inputs) (outputs "./data/weekly-incidence.csv") (values) (n= ame . "download"))'=20 --2021-03-26 13:55:22-- http://www.sentiweb.fr/datasets/incidence-PAY-3.csv Resolving www.sentiweb.fr (www.sentiweb.fr)... 134.157.220.17 Connecting to www.sentiweb.fr (www.sentiweb.fr)|134.157.220.17|:80... conne= cted. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/csv] Saving to: =E2=80=98./data/weekly-incidence.csv=E2=80=99 ./data/weekly-incid [ <=3D> ] 83,50K --.-KB/s in 0,= 008s=20=20 2021-03-26 13:55:22 (10,5 MB/s) - =E2=80=98./data/weekly-incidence.csv=E2= =80=99 saved [85509] Backtrace: 6 (primitive-load "/home/hinsen/.config/guix/current/bin/guix") In guix/ui.scm: 2164:12 5 (run-guix-command _ . _) In srfi/srfi-1.scm: 460:18 4 (fold # =E2=80=A6) 460:18 3 (fold # =E2=80=A6) In gwl/workflows.scm: 392:21 2 (_ # ()) In srfi/srfi-1.scm: 634:9 1 (for-each # =E2=80=A6) In guix/ui.scm: 566:4 0 (_ system-error "symlink" _ _ _) guix/ui.scm:566:4: In procedure symlink: File exists: "/tmp/gwl/mwmeuuhnu7s= v4mpouj7o5x4se4qp5n5auzhpkb7y7oxidoxzc6ra/./data/weekly-incidence.csv"