From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: Re: running a script in a post-build hook? Date: Thu, 13 Oct 2016 21:50:55 +0200 Message-ID: <8760owauzk.fsf@gnu.org> References: <877f9kfar7.fsf@mdc-berlin.de> <87a8egj696.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:37384) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bum1v-0001kR-29 for help-guix@gnu.org; Thu, 13 Oct 2016 15:51:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bum1r-00067V-Rc for help-guix@gnu.org; Thu, 13 Oct 2016 15:51:03 -0400 In-Reply-To: (Ricardo Wurmus's message of "Thu, 13 Oct 2016 13:33:05 +0200") List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: help-guix-bounces+gcggh-help-guix=m.gmane.org@gnu.org Sender: "Help-Guix" To: Ricardo Wurmus Cc: help-guix Hello! Ricardo Wurmus skribis: > Ludovic Court=C3=A8s writes: > >> Hello! >> >> Ricardo Wurmus skribis: >> >>> here=E2=80=99s my problem: I need to have the store on a big slow NFS s= erver >>> with online compression and deduplication. This means that *everything* >>> in Guix is slow: downloading binaries, building packages from source, >>> building a new profile generation =E2=80=94 it=E2=80=99s all *very* slo= w. >> >> This is a tricky use case=E2=80=A6 ISTR it=E2=80=99s not possible in your case, but in an ideal we=E2=80=99d b= uild on a local file system and then export it over NFS (instead of building directly on NFS). >>> Could the build hook feature be used for this, maybe by wrapping the >>> normal build such that a script is run when it finishes? >> >> The build hook =E2=80=9Cprotocol=E2=80=9D doesn=E2=80=99t work like this= . The daemon sends a >> build request, which the hook can accept, postpone, or decline (see >> (guix scripts offload)). When it accepts, the hook cannot invoke the >> daemon (it=E2=80=99s not =E2=80=9Creentrant=E2=80=9D.) The substituter = protocol is similar. > > Hmm, thanks for your input! > > Okay, so the build hook feature couldn=E2=80=99t be (ab)used for this, bu= t would > it be okay to patch the daemon to optionally run a script upon > completing a store action? OK in what sense? :-) It=E2=80=99s certainly doable. >> Otherwise maybe a file system level hack? Like making /gnu/store a >> unionfs that writes elsewhere? > > I find the file system to be the wrong level of abstraction. And > unionfs seems like a brittle solution. I=E2=80=99d much rather operate on > individual store items on demand. (The localstatedir is small enough to > copy fully each time something happens.) Yes, but we=E2=80=99re really working around the slowness of the file syste= m, so in that sense the file system is precisely the place where (bad) things are happening. > I=E2=80=99d really like to take advantage of the facts that the store is = append > only (with the exception of =E2=80=9Cguix gc=E2=80=9D) and that the daemo= n knows what=E2=80=99s > going on. A mechanism to invalidate store items that have been reclaimed would also be needed, though. Another option would be to have two full-blown stores, one on a fast file system and the other one on NFS, each with its own database and possibly guix-daemon instance. You would periodically send any missing items from the former to the latter, along the lines of: guix archive --export \ `guix gc --list-live | ssh machine-with-slow-fs guix archive --missing` \ | ssh machine-with-slow-fs guix archive --import =20=20=20 It=E2=80=99s a bit of a sledgehammer, but it would copy only the new items = to the target machine. (I plan to have a =E2=80=98guix copy=E2=80=99 command that does the 3 lines= above.) With the hook that you suggest above, you could run this synchronization command upon build completion rather than periodically. Yet another option, assuming you have two separate stores like this, would be to export the first store with =E2=80=98guix publish=E2=80=99 and = have the second one take everything from there. Thoughts? Ludo=E2=80=99.