From mboxrd@z Thu Jan 1 00:00:00 1970 From: Timothy Sample Subject: Re: Graft hooks Date: Wed, 22 Aug 2018 14:29:49 -0400 Message-ID: <87y3cy8db6.fsf@ngyro.com> References: <87k1ovbc0t.fsf@ngyro.com> <87y3d1h05e.fsf@gnu.org> <87k1ojafqa.fsf@ngyro.com> <874lfmh477.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:37217) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fsXtG-0004dJ-NI for guix-devel@gnu.org; Wed, 22 Aug 2018 14:30:00 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fsXtB-0004m3-NT for guix-devel@gnu.org; Wed, 22 Aug 2018 14:29:58 -0400 In-Reply-To: <874lfmh477.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Wed, 22 Aug 2018 16:21:48 +0200") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org Hi Ludo, ludo@gnu.org (Ludovic Court=C3=A8s) writes: > Hello Timothy, > > Timothy Sample skribis: > >> ludo@gnu.org (Ludovic Court=C3=A8s) writes: >> >>> Hello Timothy, >>> >>> Timothy Sample skribis: >>> >>>> The basic idea would be to add a field (or use a property) to the >>>> package record. Let=E2=80=99s call it =E2=80=9Cgraft-hook=E2=80=9D. = It would be Scheme code >>>> that gets run after grafting takes place, giving us a chance to patch >>>> special things like checksums. The hook would be passed the list of >>>> files that were been modified during grafting. Then, in the Racket >>>> package for example, I could write a graft-hook that updates the SHA-1 >>>> hash of each of the modified source files. >>>> >>>> Since grafting is done at the derivation level, the hook code would ha= ve >>>> to be propagated down from the package level. I haven=E2=80=99t looke= d at all >>>> the details yet, because maybe this is a bad idea and I shouldn=E2=80= =99t waste >>>> my time! :) My first impression is that it is not too tricky. >>>> >>>> Are these problems too specialized to deserve a general mechanism like >>>> this? Let me know what you think! >>> >>> I agree that this would be the right thing to do! (I=E2=80=99d really = like to >>> do it for GDB as discussed in .) >>> >>> Package properties would be the right way to make it extensible, but >>> there are complications (notably we=E2=80=99d need to use gexps, but bu= ild >>> systems don=E2=80=99t use gexps yet.) >> >> But soon, right? ;) > > Well, it=E2=80=99s complicated. :-) > > Also, I realized that some things, like the .gnu_debuglink and build-id > hooks, don=E2=80=99t really fit in any package; they=E2=80=99re transvers= e. You=E2=80=99re right. Packages are the wrong place. What about build syst= ems? Maybe it makes sense (theoretically) to define graft hooks in build systems, and then modify them (if necessary) using the =E2=80=9Carguments= =E2=80=9D field of a package. Isn=E2=80=99t it the GNU or Go build systems that are ultima= tely responsible for these checksums? Shouldn=E2=80=99t it be their job to tell= the grafting code how to fix them? >> Here=E2=80=99s a draft patch (it=E2=80=99s mercifully small). I have a = few questions >> about it, but if it looks like the right approach, I will clean it up >> and submit it. >> >> Basically, it checks if we are grafting Racket, and then adds some code >> to the build expression to run the hook. > > OK. In theory, should it be just for Racket, or should it also be for > Racket libraries (we don=E2=80=99t have any currently AFAIK)? I could imagine attaching the hook to our (theoretical) Racket build system. ;) In the meantime, we can check if =E2=80=9Cshare/racket=E2=80= =9D exists in the output and run the code then. Of course, if it were a Racket library (and not Racket itself), I wouldn=E2=80=99t be able to use =E2=80= =9Craco setup=E2=80=9D, because I wouldn=E2=80=99t be able to find it. >> Also, is there a preference for patching the files using Guile or using >> an external tool? This patch uses Racket=E2=80=99s =E2=80=9Craco setup= =E2=80=9D command to >> recompile the files and fix the checksums. Unfortunately, it also >> updates timestamps. I=E2=80=99m pretty sure our Racket package is not >> reproducible at the moment, so I didn=E2=80=99t worry about it too much.= The >> timestamps could be patched out, though. The reason I shied away from >> writing my own code is that Racket also hashes all the dependencies for >> a bytecode file. This means that the custom code would have to traverse >> the Racket dependency graph to get the checksums right. It is not too >> hard to do so, but it would be a couple hundred lines of code (compared >> to the five or so it took to invoke =E2=80=9Craco setup=E2=80=9D). > > Regarding whether or not to write our own code: let=E2=80=99s do whicheve= r is > more convenient. In this case, using =E2=80=98raco setup=E2=80=99 looks = like the right > thing to do, given that raco is available in the build environment > anyway (see below); for .gnu_debuglink, I found it nicer (and more fun > :-)) to write a Guile module. > > Regarding timestamps: I guess there=E2=80=99s no problem since timestamps= are > reset in the store. Whoops! I didn=E2=80=99t mean the file metadata, I meant in the bytecode f= iles themselves. Also, I only saw the changes in diffoscope and assumed they were timestamps. I looked more closely and realized they were more checksums that I didn=E2=80=99t notice before (it should have been obvious because they are 20 bytes...). They are supposed to be updated. Nothing to see here. :) > Some comments: > >> diff --git a/guix/grafts.scm b/guix/grafts.scm >> index d6b0e93e8..88a99312d 100644 >> --- a/guix/grafts.scm >> +++ b/guix/grafts.scm >> @@ -75,6 +75,36 @@ >> (($ (? string? item)) >> item))) >>=20=20 >> +(define (fix-racket-checksums store drv system) >> + (define racket-drv >> + (let ((package-derivation (module-ref (resolve-interface '(guix pac= kages)) >> + 'package-derivation)) >> + (racket (module-ref (resolve-interface '(gnu packages scheme)) >> + 'racket))) >> + (package-derivation store racket system #:graft? #f))) >> + >> + (define hook-exp >> + `(lambda (input output mapping) >> + (let ((raco (string-append output "/bin/raco"))) >> + ;; Setting PLT_COMPILED_FILE_CHECK to "exists" tells Racket to >> + ;; ignore timestamps when checking if a compiled file is valid. >> + ;; Without it, Racket attempts a complete rebuild of >> + ;; everything. >> + (setenv "PLT_COMPILED_FILE_CHECK" "exists") >> + ;; All of the --no-* flags below keep Racket from making >> + ;; unecessary and unhelpful changes (like rewriting scripts and >> + ;; reverting their shebangs in the process). >> + (invoke raco "setup" "--no-launcher" "--no-install" >> + "--no-post-install" "--no-info-domain" "--no-docs")))) > > Since this is used when grafting Racket, I would suggest moving this > graft to the =E2=80=9Cbuild side=E2=80=9D entirely, similar to what I did= in > . Probably > you=E2=80=99d just add a single procedure to (guix build graft) and add i= t to > %graft-hooks. > > That procedure could be the same as what you have above, except that > it=E2=80=99d run OUT/bin/raco, if it exists, and do nothing if OUT/bin/ra= co does > not exist. > > WDYT? This makes sense. Following my note above, I think I will try and finish my update code in Guile, and then use the existence of =E2=80=9Cshare/racket=E2=80=9D as the = heuristic to determine if it should run. Maybe I will package a Racket library, too, so I can test it. > Thanks, > Ludo=E2=80=99. Thanks for the feedback! -- Tim