From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ricardo Wurmus Subject: Re: running a script in a post-build hook? Date: Thu, 13 Oct 2016 13:33:05 +0200 Message-ID: References: <877f9kfar7.fsf@mdc-berlin.de> <87a8egj696.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:41961) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bueGJ-0001VO-FX for help-guix@gnu.org; Thu, 13 Oct 2016 07:33:26 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bueGF-0001EB-9d for help-guix@gnu.org; Thu, 13 Oct 2016 07:33:22 -0400 In-Reply-To: <87a8egj696.fsf@gnu.org> List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: help-guix-bounces+gcggh-help-guix=m.gmane.org@gnu.org Sender: "Help-Guix" To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: help-guix Ludovic Court=C3=A8s writes: > Hello! > > Ricardo Wurmus skribis: > >> here=E2=80=99s my problem: I need to have the store on a big slow NFS = server >> with online compression and deduplication. This means that *everythin= g* >> in Guix is slow: downloading binaries, building packages from source, >> building a new profile generation =E2=80=94 it=E2=80=99s all *very* sl= ow. > > This is a tricky use case=E2=80=A6 > >> So I thought: I should be able to just have the store on fast local >> disks and copy over store items to the slow NFS server when they are >> done. This is not easy because the store is very big and at the >> filesystem level we cannot take advantage of the fact that the store i= s >> append-only for the most part. So running rsync, for example, won=E2=80= =99t cut >> it. >> >> The daemon, however, does have information about completed builds and >> new store items. Does the daemon have a post-build hook that can be >> called with the names of the new store items which I could then copy >> over? (Or even the names of store items I need to delete after =E2=80= =9Cguix >> gc=E2=80=9D.) >> >> Could the build hook feature be used for this, maybe by wrapping the >> normal build such that a script is run when it finishes? > > The build hook =E2=80=9Cprotocol=E2=80=9D doesn=E2=80=99t work like thi= s. The daemon sends a > build request, which the hook can accept, postpone, or decline (see > (guix scripts offload)). When it accepts, the hook cannot invoke the > daemon (it=E2=80=99s not =E2=80=9Creentrant=E2=80=9D.) The substituter= protocol is similar. Hmm, thanks for your input! Okay, so the build hook feature couldn=E2=80=99t be (ab)used for this, bu= t would it be okay to patch the daemon to optionally run a script upon completing a store action? >> Finally, is this a good idea? Or is /gnu/store/.links going to be a >> problem? Should a post-build hook also inform about deduplication >> decisions? Or should I just turn of deduplication in my case? > > Turning off deduplication would probably help because deduplication is > I/O intensive. > > Otherwise maybe a file system level hack? Like making /gnu/store a > unionfs that writes elsewhere? I find the file system to be the wrong level of abstraction. And unionfs seems like a brittle solution. I=E2=80=99d much rather operate o= n individual store items on demand. (The localstatedir is small enough to copy fully each time something happens.) I=E2=80=99d really like to take advantage of the facts that the store is = append only (with the exception of =E2=80=9Cguix gc=E2=80=9D) and that the daemo= n knows what=E2=80=99s going on. ~~ Ricardo