all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
From: Ricardo Wurmus <rekado@elephly.net>
To: Konrad Hinsen <konrad.hinsen@fastmail.net>
Cc: gwl-devel@gnu.org
Subject: Re: Managing data files in workflows
Date: Wed, 07 Apr 2021 13:38:37 +0200	[thread overview]
Message-ID: <87blaqz5mq.fsf@elephly.net> (raw)
In-Reply-To: <m1lfa1f58g.fsf@ordinateur-de-catherine--konrad.home>


Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> Looking at the source code in (gwl cache), restoring means symlinking
> the target file to the cached file, which can't work given that the
> cache is already a symlink to the target file.
>
> So... I don't understand how the cache is supposed to work. If it stores
> symlinks, there is no need to restore anything. If it is supposed to
> store copies, then that's not what it does.

Right, that’s really the heart of the problem here.  Originally, I used
hardlinks exclusively, but they don’t work everywhere.  So I added
symlinks, but obviously they have different semantics.

We can fix the problem with symlinks by restoring the target of the link
instead of the link itself, but I feel that we need to take a step back
and consider what this cache is really to be used for.

The cache assumes again that files are immutable when really they are
not guaranteed to be immutable.  Both symlinks and hardlinks don’t give
us any guarantees.

I really would like to have independent copies of input and output
files, but I also don’t want to needlessly copy files around or use up
more space than absolutely necessary.   We could punt on the problem of
optimal space consumption and simply copy files to the cache.

What do you think?

-- 
Ricardo


  reply	other threads:[~2021-04-07 11:39 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-03-25  9:57 Managing data files in workflows Konrad Hinsen
2021-03-26  7:02 ` zimoun
2021-03-26 12:46   ` Konrad Hinsen
2021-03-26  8:47 ` Ricardo Wurmus
2021-03-26 12:30   ` Konrad Hinsen
2021-03-26 12:54     ` Konrad Hinsen
2021-03-26 13:13     ` Ricardo Wurmus
2021-03-26 15:36       ` Konrad Hinsen
2021-04-01 13:27         ` Ricardo Wurmus
2021-04-02  8:41           ` Konrad Hinsen
2021-04-07 11:38             ` Ricardo Wurmus [this message]
2021-04-08  7:28               ` Konrad Hinsen
2021-05-03  9:18                 ` Ricardo Wurmus
2021-05-03 11:58                   ` zimoun
2021-05-03 13:47                     ` Ricardo Wurmus

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87blaqz5mq.fsf@elephly.net \
    --to=rekado@elephly.net \
    --cc=gwl-devel@gnu.org \
    --cc=konrad.hinsen@fastmail.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.