From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: Re: [PATCH] Prototype register-path Date: Mon, 19 Jun 2017 13:56:49 +0200 Message-ID: <8760fsxkam.fsf@gnu.org> References: <8737axodjs.fsf@cune.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:57340) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dMvIe-00049d-4L for guix-devel@gnu.org; Mon, 19 Jun 2017 07:56:57 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dMvIb-0001Yz-1m for guix-devel@gnu.org; Mon, 19 Jun 2017 07:56:56 -0400 In-Reply-To: <8737axodjs.fsf@cune.org> (Caleb Ristvedt's message of "Sun, 18 Jun 2017 04:22:47 -0500") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Caleb Ristvedt Cc: guix-devel@gnu.org Hello! Caleb Ristvedt skribis: > Well, I took a look at libstore/build.cc, and that is a lot of code. I > hope I can get away with only reading as much as is necessary to clarify > details. To that end, I'd like to make sure that my high-level > understanding of the build process is accurate. > > - A derivation is a node in a (acyclic?) graph of "what depends on what" = or, > alternately put, what is input to what. Yes, a directed acyclic graph. > - Derivations can depend on other derivations or plain files (can those f= iles be > directories?), where "plain files" cannot depend on anything else. Right, =E2=80=9Cplain files=E2=80=9D are referred to as =E2=80=9Csources=E2= =80=9D in the code (e.g., the =E2=80=98sources=E2=80=99 field of ). =E2=80=9CSources=E2=80= =9D are content-addressable things added with =E2=80=98add-text-to-store=E2=80=99, =E2=80=98add-to-stor= e=E2=80=99, and similar. > - "Building" a derivation involves first ensuring that all of its inputs = are > "built" (in the case of inputs that are derivations, this means ensurin= g its > output exists as a store item, and if not, producing it by this same > process. In the case of plain file inputs, this means just ensuring tha= t they > exist as a store item), and then preparing a build environment. The "bu= ilder" > script is then executed in a chroot rooted under the build tree. After = it has > completed, the output (where is it put during the build process?) from > building the derivation is placed in the store (and registered). Exactly. > - "preparing a build environment" involves first creating a directory in = which > the build will be run, then populating it with certain expected files a= nd > directories (as detailed in 2.4.1 of the manual). Also, the store items > produced from building the inputs need to somehow be made available (is= there > a gnu/store/ under the build tree? I feel like I don't quite understand= where > everything goes in the build tree). Also, set any environment variables > associated with the derivation. Yes. In the build environment, there=E2=80=99s the subset of /gnu/store th= at is explicitly declared as a dependency, directly or indirectly. For example, we can see it here: --8<---------------cut here---------------start------------->8--- scheme@(guile-user)> ,use(guix) scheme@(guile-user)> ,enter-store-monad store-monad@(guile-user) [1]> (gexp->derivation "contents" #~(begin (use-modules (ice-9 ftw)) (call-with-output-file #$output (lambda (port) (write (scandir "/gnu/store") port))))) $2 =3D # /gnu/store/a1z90bzb7xl6glp033jkxjc1n3w0lw7w-contents 41847d0> store-monad@(guile-user) [1]> (built-derivations (list $2)) $3 =3D #t store-monad@(guile-user) [1]> ,q scheme@(guile-user)> ,pp (call-with-input-file (derivation->output-path $2)= read) $4 =3D ("." ".." "02426nwiy32cscm4h83729vn5ws1gs2i-bash-static-4.4.12" "3lsfrwlp1qa345x71yw5w49i2mpp0vxm-guile-2.0.14" "6czmriicly50bljbp6p34py5a63wzf05-contents-builder" "9f66pvz4vf1d9k8iir6asdp3l8k58cnn-libatomic-ops-7.4.4" "a1z90bzb7xl6glp033jkxjc1n3w0lw7w-contents" "b7w6bjp602qvhbsjs535dfax8v7wy8s8-gmp-6.1.2" "dhc2iy059hi91fk55dcv79z09kp6500y-gcc-5.4.0-lib" "dj9w9y66ncmn7qpnahz5fhqjjwqrnbjm-ncurses-6.0" "hvyk1qyph1hihfmym1w271ygp84adb0v-readline-7.0" "k7029k5va68lkapbzcycdzj7m5bjb4b8-bash-4.4.12" "kndl3vllk4bdq1xd3p8k67h8jrsq8xbq-readline-7.0" "lcmcm4c0zjv3sa9amdrhaszd7vwwxjh1-libltdl-2.4.6" "px46g18zg3sjgndwdcsgm6027w7s5gbc-pkg-config-0.29.1" "q1x4v3x8v2g59d244hl7k0i1n4h83c9a-ncurses-6.0" "rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25" "v4h4qw8a95479capaq08vs4zdyxdijhv-libunistring-0.9.7" "wqx8sxqjvz323vk9xalrhqk5f35yd42f-libffi-3.2.1" "yr7m8ldp3n40mrzjax91cj9hjw1k2a58-libgc-7.6.0") --8<---------------cut here---------------end--------------->8--- What we see above is Guile and its dependencies, along with the output and its build script. > Assuming I've got the gist of it sort of right, that leaves one more ques= tion: > how are hashes for the paths computed? It's not sha256 like the database = hashes, > it seems to be 160 bits in size. How are all the inputs hashed together? = Are all > the inputs put in one big directory and that is hashed in nar form, or wh= at? And > what about store items that don't have any inputs (for example, something= placed > in there with add-text-to-store)? I suppose in that case they are their o= wn > input, perhaps? See (guix derivations) for how these hashes are computed, and also =E2=80=98store-path=E2=80=99, =E2=80=98output-path=E2=80=99, and =E2=80=98f= ixed-output-path=E2=80=99 in (guix store). Normally you won=E2=80=99t have to reimplement any of these. > Some of those answers might be in the manual, but I didn't manage to > find them. The manual doesn=E2=80=99t tell much about internal APIs. :-) HTH! Ludo=E2=80=99.