Newbie: What's the importance of *.drv, *.-builder, etc. files in /gnu/store?

Newbie: What's the importance of *.drv, *.-builder, etc. files in /gnu/store?

[Rodrigo Morales]

Table of Contents
_________________

1. TL; DR
2. The context
3. The question


1 TL; DR
========

  I'm a newbie when it comes to understanding the file extensions of
  files that exist in `/gnu/store'. I want to know what each file
  extension means and how they are important to the operating system or
  for the user. Namely, the extensions that I don't understand yet are +
  `*.drv', `*-builder', `*.tar.xz.drv.', `*.tar.xz-builder.' and +
  `*-honor-localdir.patch'. References to the manual or documentation on
  this topic are greatly appreciated.


2 The context
=============

  Initially, my `/gnu/store' have no directory whose name contain the
  word `transmission'.

  ,----
  | find /gnu/store -mindepth 1 -maxdepth 1 -name '*transmission*' | wc -l
  `----

  ,----
  | 0
  `----

  Now, I install the package `transmission'.

  ,----
  | delta% guix package -i transmission
  | The following package will be installed:
  |    transmission 3.00
  |
  | 1.5 MB will be downloaded
  |  transmission-3.00  664KiB

 222KiB/s 00:03 ▕██████████████████▏ 100.0% transmission-3.00-gui  757KiB

                                     271KiB/s 00:03 ▕██████████████████▏
100.0%The following derivation will be built:
  |   /gnu/store/6x0f3mczkz5dmsrrnw5z6nwd2l0lnr33-profile.drv
  |
  | applying 27 grafts for transmission-3.00 ...
  | building profile with 11 packages...
  | delta% echo $?
  | 0
  `----

  After installing the package, there are 4 directories and 8 files in
  `/gnu/store'. See proof below.

  ,----
  | find /gnu/store -mindepth 1 -maxdepth 1 -name '*transmission*' | wc -l
  | find /gnu/store -mindepth 1 -maxdepth 1 -name '*transmission*' -printf
'%y %p\n'
  `----

  ,----
  | 12
  | f
/gnu/store/w5axm22g9nrgbkhrsjlf4zvnksavdjwp-transmission-3.00.tar.xz.drv
  | f
/gnu/store/d4h1gkavik552f9bqcnimiylhrp18b7c-transmission-honor-localedir.patch
  | f
/gnu/store/xivc36rv3dk5qzfmf78hzrrvnp6m1crv-transmission-3.00.tar.xz-builder
  | f
/gnu/store/01h9m5lrmxhcbxxlf45z24zgmbvm6ibj-transmission-3.00.tar.xz.drv
  | f /gnu/store/asq7m7221syq0rlwiy01lhcsj54lihyv-transmission-3.00-builder
  | f /gnu/store/v8sgmbxrji7gkd4pyc2x4wik4x29qysg-transmission-3.00.drv
  | f /gnu/store/dan5qy88nzdyn9010qz0a9666bss6x4v-transmission-3.00-builder
  | f /gnu/store/pv9yvk7qzs7ifx5fkam2mqla51y12ihh-transmission-3.00.drv
  | d /gnu/store/dxyjmmrw6vkbjmm4jx17i92rnc4mmcm3-transmission-3.00
  | d /gnu/store/p4wfwg0d6qz5navc6w5gnc6gzibhiisd-transmission-3.00-gui
  | d /gnu/store/6kmyb111bpf0mx25vd981h4zqmzp02jl-transmission-3.00-gui
  | d /gnu/store/pnq464qnrr5nrmhr3avn39b22axf1na8-transmission-3.00
  `----

  The last 4 directories in the output shown above look as if they
  contain the binaries and additional files of the build process. Please
  correct me if I'm wrong.

  ,----
  | echo -
  | find /gnu/store/p4wfwg0d6qz5navc6w5gnc6gzibhiisd-transmission-3.00-gui
-mindepth 1 -maxdepth 1
  | echo -
  | find /gnu/store/6kmyb111bpf0mx25vd981h4zqmzp02jl-transmission-3.00-gui
-mindepth 1 -maxdepth 1
  | echo -
  | find /gnu/store/dxyjmmrw6vkbjmm4jx17i92rnc4mmcm3-transmission-3.00
-mindepth 1 -maxdepth 1
  | echo -
  | find /gnu/store/pnq464qnrr5nrmhr3avn39b22axf1na8-transmission-3.00
-mindepth 1 -maxdepth 1
  `----

  ,----
  | -
  | /gnu/store/p4wfwg0d6qz5navc6w5gnc6gzibhiisd-transmission-3.00-gui/bin
  | /gnu/store/p4wfwg0d6qz5navc6w5gnc6gzibhiisd-transmission-3.00-gui/share
  | /gnu/store/p4wfwg0d6qz5navc6w5gnc6gzibhiisd-transmission-3.00-gui/etc
  | -
  | /gnu/store/6kmyb111bpf0mx25vd981h4zqmzp02jl-transmission-3.00-gui/bin
  | /gnu/store/6kmyb111bpf0mx25vd981h4zqmzp02jl-transmission-3.00-gui/share
  | /gnu/store/6kmyb111bpf0mx25vd981h4zqmzp02jl-transmission-3.00-gui/etc
  | -
  | /gnu/store/dxyjmmrw6vkbjmm4jx17i92rnc4mmcm3-transmission-3.00/bin
  | /gnu/store/dxyjmmrw6vkbjmm4jx17i92rnc4mmcm3-transmission-3.00/share
  | /gnu/store/dxyjmmrw6vkbjmm4jx17i92rnc4mmcm3-transmission-3.00/etc
  | /gnu/store/dxyjmmrw6vkbjmm4jx17i92rnc4mmcm3-transmission-3.00/lib
  | -
  | /gnu/store/pnq464qnrr5nrmhr3avn39b22axf1na8-transmission-3.00/bin
  | /gnu/store/pnq464qnrr5nrmhr3avn39b22axf1na8-transmission-3.00/share
  | /gnu/store/pnq464qnrr5nrmhr3avn39b22axf1na8-transmission-3.00/etc
  | /gnu/store/pnq464qnrr5nrmhr3avn39b22axf1na8-transmission-3.00/lib
  `----

  As I'm not very familiar with Guix build process and generations, I'm
  wondering what's the relevance of the files that are created in
  `/gnu/store' and whose file extensions are

  + `*.drv'
  + `*-builder'
  + `*.tar.xz.drv.'
  + `*.tar.xz-builder.'
  + `*-honor-localdir.patch'


3 The question
==============

  Is there any section in the Guix manual that explains the meaning of
  each extension and how they are important to the operating system or
  useful to the user? If not, could anyone briefly describe those file
  extensions please?

Re: Newbie: What's the importance of *.drv, *.-builder, etc. files in /gnu/store?

[Julien Lepiller]

Going step by step, Guix performs the following.

First, it needs to download the original sources of the package. When creating a store item, most of the time, the guix daemon is involved.

The daemon's purpose is to take a lower-level representation of a build procedure and turn that into an isolated build environment. This is called a *derivation* and it is stored in the store as a `drv` file. For downloading a file, the derivation simply contains the URL to download and the daemon performs the download itself, so no isolated environment is required here.

Then, the package contains a directive that informs Guix there is some sort of change to make in the source. There is an issue in the upstream sources, and we need to apply a patch. A `.patch` file is a standard file used in many projects to represent one or more changes to a file or a directory. It's not specific to Guix.

To apply the file, the daemon needs to create an isolated build environment where it can apply the patch. Although a derivation can specify how to make a build environment (it specifies inputs (dependencies), the type of system, expected output, etc), it does not say *what* to do with it.

The guix daemon does not run inside the isolated build environment (the "build-side"), but it needs to run something in it to get things done, somehow. This is the role of the builder, which is created by Guix and is derived from the package definition. It's a standalone Guile script that is executed on the build-side.

On the build-side, the builder extracts the original archive, applies the patch and archives the result as a new `.tar.xz` file.

Now, Guix is ready to build transmission. To do so, it creates a new derivation `.drv` that instructs the daemon how to make the isolated environment for building it, and a builder `-builder` that the daemon will execute inside the environment. The build results in the transmission outputs.

To summarize, Guix needs to:

- Download original sources, which creates a derivation (drv) and results in downloading a tar.xz to the store.
- Put a patch (.patch) in the store from its own sources, which doesn't create a derivation. This is because a derivation can only refer to another store item.
- Apply the patch to the sources, which creates a derivation (.drv) and a builder (tar.xz-builder) that result in creating a modified tar.xz in the store.
- Build the transmission package, which creates a derivation (drv) and a builder (-builder) that result in the transmission outputs you wanted in the first place.

HTH!

Re: Newbie: What's the importance of *.drv, *.-builder, etc. files in /gnu/store?

[(]

[-- Attachment #1: Type: text/plain, Size: 390 bytes --]

On Sat Apr 8, 2023 at 1:49 AM BST, Rodrigo Morales wrote:
>   Is there any section in the Guix manual that explains the meaning of
>   each extension and how they are important to the operating system or
>   useful to the user? If not, could anyone briefly describe those file
>   extensions please?

https://guix.gnu.org/en/blog/2023/dissecting-guix-part-1-derivations/

    -- (

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]