Offline build failure

Offline build failure

[Greg Hogan]

[-- Attachment #1: Type: text/plain, Size: 3285 bytes --]

I am attempting an offline build without success. I have a Guix 1.2.0 node
with internet access on which I download sources with transitive
dependencies:
  $ guix build --sources=transitive tzdata > ~/transfer

I then copy the files as root to a Guix 1.2.0 node without internet access
(only local network access):
  # cat /home/<user>/transfer | xargs -n 1 -I{} scp -p {} <ip>:{}

I then attempt to build on the offline node:
  $ guix build --no-substitutes tzdata

Guix starts downloading and the transferred file is gone! I'm lost as to
why a new download attempt is made as the file data and timestamps match
the original server.

----------------------------------------
The following derivations will be built:
   /gnu/store/83aqdbhnk33p6phd92b48aqb2wlhr7fa-tzdata-2020a.drv
   /gnu/store/0j6gi10v9hz2s2413nhwbzihrydafgvj-glibc-2.31.drv
   /gnu/store/04s4qih73dwvpzpvxs299wcg9dam8f0y-bash-mesboot-4.4.drv
<cut>
   /gnu/store/xxwjr96ihqhpv7ag4cbwzk0r0p5cxakk-tzdata2020a.tar.gz.drv
building
/gnu/store/m5z2bvv96wqhxs4aahjp5xxm910czkhl-tzcode2020a.tar.gz.drv...

Starting download of
/gnu/store/r7xqxgfmm5msvazlzxaccw7h0h6dm898-tzcode2020a.tar.gz
From https://data.iana.org/time-zones/releases/tzcode2020a.tar.gz...
----------------------------------------

Building on the original, internet-connected server no re-download is
attempted, though additional packages not included in the transitive
sources are requested.

----------------------------------------
The following derivations will be built:
   /gnu/store/83aqdbhnk33p6phd92b48aqb2wlhr7fa-tzdata-2020a.drv
   /gnu/store/0j6gi10v9hz2s2413nhwbzihrydafgvj-glibc-2.31.drv
   /gnu/store/04s4qih73dwvpzpvxs299wcg9dam8f0y-bash-mesboot-4.4.drv
<cut>
   /gnu/store/69z1j6lx09krsa019a2r9v72i80lnkql-libsigsegv-2.12.drv
   /gnu/store/0npjqnxv0acwdkxnc6askz2ac4dikj73-libsigsegv-2.12.tar.gz.drv
building
/gnu/store/4lqmlyvbvddzi50cbqjwh8lzb2lvmkia-module-import-compiled.drv...
successfully built
/gnu/store/4lqmlyvbvddzi50cbqjwh8lzb2lvmkia-module-import-compiled.drv
building
/gnu/store/4d3cjvjrgs0q38fs3bgfajcw7apc0vml-Python-3.5.9.tar.xz.drv...

Starting download of
/gnu/store/f99fblkzb6ip268sg096shhs7wzjyp55-Python-3.5.9.tar.xz
From https://www.python.org/ftp/python/3.5.9/Python-3.5.9.tar.xz...
downloading from https://www.python.org/ftp/python/3.5.9/Python-3.5.9.tar.xz
...
 Python-3.5.9.tar.xz  14.7MiB

               141.9MiB/s 00:00 [##################] 100.0%
successfully built
/gnu/store/4d3cjvjrgs0q38fs3bgfajcw7apc0vml-Python-3.5.9.tar.xz.drv
building
/gnu/store/ldkjb6wjcl6dbf0glpfw9f5wh1ia8bq3-bash-2.05b.tar.gz.drv...

Starting download of
/gnu/store/zkkcabiqcy11wy4wkn8bysava6qz8w7v-bash-2.05b.tar.gz
From https://ftpmirror.gnu.org/gnu/bash/bash-2.05b.tar.gz...
following redirection to `https://gnu.askapache.com/bash/bash-2.05b.tar.gz'.
..
downloading from https://ftpmirror.gnu.org/gnu/bash/bash-2.05b.tar.gz ...
 bash-2.05b.tar.gz  1.9MiB

                  4.1MiB/s 00:00 [##################] 100.0%
successfully built
/gnu/store/ldkjb6wjcl6dbf0glpfw9f5wh1ia8bq3-bash-2.05b.tar.gz.drv
building /gnu/store/8nnmv6hfwg543s7b0sbx54ysi514dsz6-bash-4.4.tar.gz.drv...

<cut>
------------------------------

If there is a better way to setup / configure / execute offline builds
please let me know!

Thanks,
Greg

[-- Attachment #2: Type: text/html, Size: 4631 bytes --]

Re: Offline build failure

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 1543 bytes --]

Hullo Greg,

Greg Hogan 写道：
> If there is a better way to setup / configure / execute offline 
> builds
> please let me know!

...yes :-)

> I am attempting an offline build without success. I have a Guix 
> 1.2.0 node
> with internet access on which I download sources with transitive
> dependencies:
>   $ guix build --sources=transitive tzdata > ~/transfer

OK.

> I then copy the files as root to a Guix 1.2.0 node without 
> internet access
> (only local network access):
>   # cat /home/<user>/transfer | xargs -n 1 -I{} scp -p {} 
>   <ip>:{}

Now you've basically reinvented ‘guix copy --to=<ip>’, but in a 
way that won't update the store database in /var/guix/db.  I'm 
afraid that won't work.

Guix won't ‘see’ the files you copy to the remote store and will 
consider them G to be C'd next time you run ‘guix gc’.  Or in this 
case:

> Guix starts downloading and the transferred file is gone!

Same thing.

> I'm lost as to
> why a new download attempt is made as the file data and 
> timestamps match
> the original server.

If the file isn't registered in the database, the store item is 
never considered valid.  Guix doesn't (yet) care about the 
data/timestamps at this point.

If there's a reason you can't/won't use ‘guix copy’, you might 
work around that by copying each file in ~/transfer to, say, 
<ip>:/tmp/staging (instead of <ip>:/gnu/store), then running ‘guix 
download /tmp/staging/<file>...’ on the remote host.

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: Offline build failure

[Greg Hogan]

[-- Attachment #1: Type: text/plain, Size: 3477 bytes --]

Thanks, Tobias. I am now properly populating the store. I have switched to
using 'guix graph --type=derivation' to pull in what seems to be the full
set of dependencies. I am seeing a strange issue when importing a package.
I can typically duplicate and import a file and it is copied to the
original location, for example:

----------------------------------------
$ cp -a /gnu/store/v2zls012iwxyw6058cir7n2b792lsvc9-perl-5.30.2.tar.gz
perl-5.30.2.tar.gz
$ guix download perl-5.30.2.tar.gz
/gnu/store/v2zls012iwxyw6058cir7n2b792lsvc9-perl-5.30.2.tar.gz
128nfdxcvxfn5kq55qcfrx2851ys8hv794dcdxbyny8rm7w7vnv6

$ cp -a /gnu/store/17acz7ks1f7xn6yp1a2y4g7vc6bhr8wc-make-3.82.tar.gz
make-3.82.tar.gz
$ guix download make-3.82.tar.gz
/gnu/store/17acz7ks1f7xn6yp1a2y4g7vc6bhr8wc-make-3.82.tar.gz
1rs2f9hmvy3q6zkl15jnlmnpgffm0bhw5ax0h5c7q604wqrip69x
----------------------------------------

For the Python-3.5.9 tarball the first version loads in the expected way.
The second tarball is restored to a new location:

----------------------------------------
$ cp -a /gnu/store/f99fblkzb6ip268sg096shhs7wzjyp55-Python-3.5.9.tar.xz
Python-3.5.9.tar.xz
$ guix download Python-3.5.9.tar.xz
/gnu/store/f99fblkzb6ip268sg096shhs7wzjyp55-Python-3.5.9.tar.xz
0jdh9pvx6m6lfz2liwvvhn7vks7qrysqgwn517fkpxb77b33fjn2

$ cp -a /gnu/store/nj79fxxl5wvnq7jpj2wgbx0591gkjw41-Python-3.5.9.tar.xz
Python-3.5.9.tar.xz
$ guix download Python-3.5.9.tar.xz
/gnu/store/9sa83nyjlm5dyhwys4imm1wa40mjaw1x-Python-3.5.9.tar.xz
0rkn451qfz3gbni57la00a5fbgish9jmm5bmhmgmf223vxwya447
----------------------------------------

Since the tarball is not restored to the original location the guix build
command still attempts the download and fails the offline build.

Greg Hogan

On Fri, Dec 11, 2020 at 9:42 PM Tobias Geerinckx-Rice <me@tobias.gr> wrote:

> Hullo Greg,
>
> Greg Hogan 写道：
> > If there is a better way to setup / configure / execute offline
> > builds
> > please let me know!
>
> ...yes :-)
>
> > I am attempting an offline build without success. I have a Guix
> > 1.2.0 node
> > with internet access on which I download sources with transitive
> > dependencies:
> >   $ guix build --sources=transitive tzdata > ~/transfer
>
> OK.
>
> > I then copy the files as root to a Guix 1.2.0 node without
> > internet access
> > (only local network access):
> >   # cat /home/<user>/transfer | xargs -n 1 -I{} scp -p {}
> >   <ip>:{}
>
> Now you've basically reinvented ‘guix copy --to=<ip>’, but in a
> way that won't update the store database in /var/guix/db.  I'm
> afraid that won't work.
>
> Guix won't ‘see’ the files you copy to the remote store and will
> consider them G to be C'd next time you run ‘guix gc’.  Or in this
> case:
>
> > Guix starts downloading and the transferred file is gone!
>
> Same thing.
>
> > I'm lost as to
> > why a new download attempt is made as the file data and
> > timestamps match
> > the original server.
>
> If the file isn't registered in the database, the store item is
> never considered valid.  Guix doesn't (yet) care about the
> data/timestamps at this point.
>
> If there's a reason you can't/won't use ‘guix copy’, you might
> work around that by copying each file in ~/transfer to, say,
> <ip>:/tmp/staging (instead of <ip>:/gnu/store), then running ‘guix
> download /tmp/staging/<file>...’ on the remote host.
>
> Kind regards,
>
> T G-R
>

[-- Attachment #2: Type: text/html, Size: 4266 bytes --]

Re: Offline build failure

[Mark H Weaver]

Hi Greg,

Greg Hogan <code@greghogan.com> writes:

> For the Python-3.5.9 tarball the first version loads in the expected way.
> The second tarball is restored to a new location:
>
> ----------------------------------------
> $ cp -a /gnu/store/f99fblkzb6ip268sg096shhs7wzjyp55-Python-3.5.9.tar.xz
> Python-3.5.9.tar.xz
> $ guix download Python-3.5.9.tar.xz
> /gnu/store/f99fblkzb6ip268sg096shhs7wzjyp55-Python-3.5.9.tar.xz
> 0jdh9pvx6m6lfz2liwvvhn7vks7qrysqgwn517fkpxb77b33fjn2
>
> $ cp -a /gnu/store/nj79fxxl5wvnq7jpj2wgbx0591gkjw41-Python-3.5.9.tar.xz
> Python-3.5.9.tar.xz
> $ guix download Python-3.5.9.tar.xz
> /gnu/store/9sa83nyjlm5dyhwys4imm1wa40mjaw1x-Python-3.5.9.tar.xz
> 0rkn451qfz3gbni57la00a5fbgish9jmm5bmhmgmf223vxwya447
> ----------------------------------------
>
> Since the tarball is not restored to the original location the guix build
> command still attempts the download and fails the offline build.

guix download will only work properly for the outputs of "fixed-output"
derivations, i.e. derivations for which Guix knows, in advance, the
sha256 hash of the resulting output.  This roughly corresponds to store
items that either downloaded or imported directly into the store from
local files (e.g. patches).

It will *not* work properly for the outputs of derivations that are
computed somehow, where Guix does *not* know in advance the hash of the
resulting output.  In particular, it won't work properly for outputs
that result from applying patches or snippets.

My guess is that the second tarball above was the result of applying
patches to the downloaded tarball.

To copy arbitrary store items from one machine to another, you need
"guix copy" or "guix archive".

      Regards,
        Mark