Re: Managing data files in workflows

[Konrad Hinsen <konrad.hinsen@fastmail.net>]

Ricardo Wurmus <rekado@elephly.net> writes:

> In my test case this file exists:
>
>     /tmp/gwl/lf6uca7zcyyldkcrxn3zwc275ax3ip676aqgjo75ybwojtl4emoq/data/weekly-incidence.csv

And that's the one that causes the error message for me when I run the
workflow a second time (see below). But as I understand now, the mistake
happens earlier, as this step shouldn't be executed at all.

> At least that happens in my case.  I wonder why it doesn’t work in your
> case.

Is there anything I can do to debug this?

> Yes, that’s the way it’s supposed to work already.  GWL computes the
> hashes of each chain of processes, which includes the generated process
> script, its inputs, and the hashes of all processes that lead up to this
> process.  Any change in the chain will lead to a new hash and thus a
> cache miss, leading GWL to recompute.

Excellent, that's what I was hoping to happen, given the Guix
background.

Cheers,
  Konrad.


$ guix workflow run test.w
info: Loading workflow file `test.w'...
info: Computing workflow `influenza-incidence'...
run: Executing: /bin/sh -c /gnu/store/km977swwhqj2n1mg34fq6sv4a41iabkm-gwl-download.scm '((inputs) (outputs "./data/weekly-incidence.csv") (values) (name . "download"))' 
--2021-03-26 13:55:22--  http://www.sentiweb.fr/datasets/incidence-PAY-3.csv
Resolving www.sentiweb.fr (www.sentiweb.fr)... 134.157.220.17
Connecting to www.sentiweb.fr (www.sentiweb.fr)|134.157.220.17|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/csv]
Saving to: ‘./data/weekly-incidence.csv’

./data/weekly-incid     [ <=>                ]  83,50K  --.-KB/s    in 0,008s  

2021-03-26 13:55:22 (10,5 MB/s) - ‘./data/weekly-incidence.csv’ saved [85509]

Backtrace:
           6 (primitive-load "/home/hinsen/.config/guix/current/bin/guix")
In guix/ui.scm:
  2164:12  5 (run-guix-command _ . _)
In srfi/srfi-1.scm:
   460:18  4 (fold #<procedure 7f33df87ea80 at gwl/workflows.scm:388:2 (ite…> …)
   460:18  3 (fold #<procedure 7f33df87ea60 at gwl/workflows.scm:391:13 (pr…> …)
In gwl/workflows.scm:
   392:21  2 (_ #<process download> ())
In srfi/srfi-1.scm:
    634:9  1 (for-each #<procedure 7f33df87e460 at gwl/workflows.scm:589:28…> …)
In guix/ui.scm:
    566:4  0 (_ system-error "symlink" _ _ _)

guix/ui.scm:566:4: In procedure symlink: File exists: "/tmp/gwl/mwmeuuhnu7sv4mpouj7o5x4se4qp5n5auzhpkb7y7oxidoxzc6ra/./data/weekly-incidence.csv"