From: Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>
To: "Ludovic Courtès" <ludo@gnu.org>
Cc: help-guix <help-guix@gnu.org>
Subject: Re: running a script in a post-build hook?
Date: Thu, 13 Oct 2016 13:33:05 +0200 [thread overview]
Message-ID: <idja8e8xz4e.fsf@bimsb-sys02.mdc-berlin.net> (raw)
In-Reply-To: <87a8egj696.fsf@gnu.org>
Ludovic Courtès <ludo@gnu.org> writes:
> Hello!
>
> Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:
>
>> here’s my problem: I need to have the store on a big slow NFS server
>> with online compression and deduplication. This means that *everything*
>> in Guix is slow: downloading binaries, building packages from source,
>> building a new profile generation — it’s all *very* slow.
>
> This is a tricky use case…
>
>> So I thought: I should be able to just have the store on fast local
>> disks and copy over store items to the slow NFS server when they are
>> done. This is not easy because the store is very big and at the
>> filesystem level we cannot take advantage of the fact that the store is
>> append-only for the most part. So running rsync, for example, won’t cut
>> it.
>>
>> The daemon, however, does have information about completed builds and
>> new store items. Does the daemon have a post-build hook that can be
>> called with the names of the new store items which I could then copy
>> over? (Or even the names of store items I need to delete after “guix
>> gc”.)
>>
>> Could the build hook feature be used for this, maybe by wrapping the
>> normal build such that a script is run when it finishes?
>
> The build hook “protocol” doesn’t work like this. The daemon sends a
> build request, which the hook can accept, postpone, or decline (see
> (guix scripts offload)). When it accepts, the hook cannot invoke the
> daemon (it’s not “reentrant”.) The substituter protocol is similar.
Hmm, thanks for your input!
Okay, so the build hook feature couldn’t be (ab)used for this, but would
it be okay to patch the daemon to optionally run a script upon
completing a store action?
>> Finally, is this a good idea? Or is /gnu/store/.links going to be a
>> problem? Should a post-build hook also inform about deduplication
>> decisions? Or should I just turn of deduplication in my case?
>
> Turning off deduplication would probably help because deduplication is
> I/O intensive.
>
> Otherwise maybe a file system level hack? Like making /gnu/store a
> unionfs that writes elsewhere?
I find the file system to be the wrong level of abstraction. And
unionfs seems like a brittle solution. I’d much rather operate on
individual store items on demand. (The localstatedir is small enough to
copy fully each time something happens.)
I’d really like to take advantage of the facts that the store is append
only (with the exception of “guix gc”) and that the daemon knows what’s
going on.
~~ Ricardo
next prev parent reply other threads:[~2016-10-13 11:33 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-10-07 15:18 running a script in a post-build hook? Ricardo Wurmus
2016-10-07 19:42 ` Ludovic Courtès
2016-10-13 11:33 ` Ricardo Wurmus [this message]
2016-10-13 19:50 ` Ludovic Courtès
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=idja8e8xz4e.fsf@bimsb-sys02.mdc-berlin.net \
--to=ricardo.wurmus@mdc-berlin.de \
--cc=help-guix@gnu.org \
--cc=ludo@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).