unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
* Offline build failure
@ 2020-12-11 21:11 Greg Hogan
  2020-12-11 21:42 ` Tobias Geerinckx-Rice
  0 siblings, 1 reply; 4+ messages in thread
From: Greg Hogan @ 2020-12-11 21:11 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 3285 bytes --]

I am attempting an offline build without success. I have a Guix 1.2.0 node
with internet access on which I download sources with transitive
dependencies:
  $ guix build --sources=transitive tzdata > ~/transfer

I then copy the files as root to a Guix 1.2.0 node without internet access
(only local network access):
  # cat /home/<user>/transfer | xargs -n 1 -I{} scp -p {} <ip>:{}

I then attempt to build on the offline node:
  $ guix build --no-substitutes tzdata

Guix starts downloading and the transferred file is gone! I'm lost as to
why a new download attempt is made as the file data and timestamps match
the original server.

----------------------------------------
The following derivations will be built:
   /gnu/store/83aqdbhnk33p6phd92b48aqb2wlhr7fa-tzdata-2020a.drv
   /gnu/store/0j6gi10v9hz2s2413nhwbzihrydafgvj-glibc-2.31.drv
   /gnu/store/04s4qih73dwvpzpvxs299wcg9dam8f0y-bash-mesboot-4.4.drv
<cut>
   /gnu/store/xxwjr96ihqhpv7ag4cbwzk0r0p5cxakk-tzdata2020a.tar.gz.drv
building
/gnu/store/m5z2bvv96wqhxs4aahjp5xxm910czkhl-tzcode2020a.tar.gz.drv...

Starting download of
/gnu/store/r7xqxgfmm5msvazlzxaccw7h0h6dm898-tzcode2020a.tar.gz
From https://data.iana.org/time-zones/releases/tzcode2020a.tar.gz...
----------------------------------------

Building on the original, internet-connected server no re-download is
attempted, though additional packages not included in the transitive
sources are requested.

----------------------------------------
The following derivations will be built:
   /gnu/store/83aqdbhnk33p6phd92b48aqb2wlhr7fa-tzdata-2020a.drv
   /gnu/store/0j6gi10v9hz2s2413nhwbzihrydafgvj-glibc-2.31.drv
   /gnu/store/04s4qih73dwvpzpvxs299wcg9dam8f0y-bash-mesboot-4.4.drv
<cut>
   /gnu/store/69z1j6lx09krsa019a2r9v72i80lnkql-libsigsegv-2.12.drv
   /gnu/store/0npjqnxv0acwdkxnc6askz2ac4dikj73-libsigsegv-2.12.tar.gz.drv
building
/gnu/store/4lqmlyvbvddzi50cbqjwh8lzb2lvmkia-module-import-compiled.drv...
successfully built
/gnu/store/4lqmlyvbvddzi50cbqjwh8lzb2lvmkia-module-import-compiled.drv
building
/gnu/store/4d3cjvjrgs0q38fs3bgfajcw7apc0vml-Python-3.5.9.tar.xz.drv...

Starting download of
/gnu/store/f99fblkzb6ip268sg096shhs7wzjyp55-Python-3.5.9.tar.xz
From https://www.python.org/ftp/python/3.5.9/Python-3.5.9.tar.xz...
downloading from https://www.python.org/ftp/python/3.5.9/Python-3.5.9.tar.xz
...
 Python-3.5.9.tar.xz  14.7MiB

               141.9MiB/s 00:00 [##################] 100.0%
successfully built
/gnu/store/4d3cjvjrgs0q38fs3bgfajcw7apc0vml-Python-3.5.9.tar.xz.drv
building
/gnu/store/ldkjb6wjcl6dbf0glpfw9f5wh1ia8bq3-bash-2.05b.tar.gz.drv...

Starting download of
/gnu/store/zkkcabiqcy11wy4wkn8bysava6qz8w7v-bash-2.05b.tar.gz
From https://ftpmirror.gnu.org/gnu/bash/bash-2.05b.tar.gz...
following redirection to `https://gnu.askapache.com/bash/bash-2.05b.tar.gz'.
..
downloading from https://ftpmirror.gnu.org/gnu/bash/bash-2.05b.tar.gz ...
 bash-2.05b.tar.gz  1.9MiB

                  4.1MiB/s 00:00 [##################] 100.0%
successfully built
/gnu/store/ldkjb6wjcl6dbf0glpfw9f5wh1ia8bq3-bash-2.05b.tar.gz.drv
building /gnu/store/8nnmv6hfwg543s7b0sbx54ysi514dsz6-bash-4.4.tar.gz.drv...

<cut>
------------------------------

If there is a better way to setup / configure / execute offline builds
please let me know!

Thanks,
Greg

[-- Attachment #2: Type: text/html, Size: 4631 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Offline build failure
  2020-12-11 21:11 Offline build failure Greg Hogan
@ 2020-12-11 21:42 ` Tobias Geerinckx-Rice
  2020-12-14 19:33   ` Greg Hogan
  0 siblings, 1 reply; 4+ messages in thread
From: Tobias Geerinckx-Rice @ 2020-12-11 21:42 UTC (permalink / raw)
  To: Greg Hogan; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 1543 bytes --]

Hullo Greg,

Greg Hogan 写道:
> If there is a better way to setup / configure / execute offline 
> builds
> please let me know!

...yes :-)

> I am attempting an offline build without success. I have a Guix 
> 1.2.0 node
> with internet access on which I download sources with transitive
> dependencies:
>   $ guix build --sources=transitive tzdata > ~/transfer

OK.

> I then copy the files as root to a Guix 1.2.0 node without 
> internet access
> (only local network access):
>   # cat /home/<user>/transfer | xargs -n 1 -I{} scp -p {} 
>   <ip>:{}

Now you've basically reinvented ‘guix copy --to=<ip>’, but in a 
way that won't update the store database in /var/guix/db.  I'm 
afraid that won't work.

Guix won't ‘see’ the files you copy to the remote store and will 
consider them G to be C'd next time you run ‘guix gc’.  Or in this 
case:

> Guix starts downloading and the transferred file is gone!

Same thing.

> I'm lost as to
> why a new download attempt is made as the file data and 
> timestamps match
> the original server.

If the file isn't registered in the database, the store item is 
never considered valid.  Guix doesn't (yet) care about the 
data/timestamps at this point.

If there's a reason you can't/won't use ‘guix copy’, you might 
work around that by copying each file in ~/transfer to, say, 
<ip>:/tmp/staging (instead of <ip>:/gnu/store), then running ‘guix 
download /tmp/staging/<file>...’ on the remote host.

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Offline build failure
  2020-12-11 21:42 ` Tobias Geerinckx-Rice
@ 2020-12-14 19:33   ` Greg Hogan
  2020-12-15  2:47     ` Mark H Weaver
  0 siblings, 1 reply; 4+ messages in thread
From: Greg Hogan @ 2020-12-14 19:33 UTC (permalink / raw)
  To: Tobias Geerinckx-Rice; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 3477 bytes --]

Thanks, Tobias. I am now properly populating the store. I have switched to
using 'guix graph --type=derivation' to pull in what seems to be the full
set of dependencies. I am seeing a strange issue when importing a package.
I can typically duplicate and import a file and it is copied to the
original location, for example:

----------------------------------------
$ cp -a /gnu/store/v2zls012iwxyw6058cir7n2b792lsvc9-perl-5.30.2.tar.gz
perl-5.30.2.tar.gz
$ guix download perl-5.30.2.tar.gz
/gnu/store/v2zls012iwxyw6058cir7n2b792lsvc9-perl-5.30.2.tar.gz
128nfdxcvxfn5kq55qcfrx2851ys8hv794dcdxbyny8rm7w7vnv6

$ cp -a /gnu/store/17acz7ks1f7xn6yp1a2y4g7vc6bhr8wc-make-3.82.tar.gz
make-3.82.tar.gz
$ guix download make-3.82.tar.gz
/gnu/store/17acz7ks1f7xn6yp1a2y4g7vc6bhr8wc-make-3.82.tar.gz
1rs2f9hmvy3q6zkl15jnlmnpgffm0bhw5ax0h5c7q604wqrip69x
----------------------------------------

For the Python-3.5.9 tarball the first version loads in the expected way.
The second tarball is restored to a new location:

----------------------------------------
$ cp -a /gnu/store/f99fblkzb6ip268sg096shhs7wzjyp55-Python-3.5.9.tar.xz
Python-3.5.9.tar.xz
$ guix download Python-3.5.9.tar.xz
/gnu/store/f99fblkzb6ip268sg096shhs7wzjyp55-Python-3.5.9.tar.xz
0jdh9pvx6m6lfz2liwvvhn7vks7qrysqgwn517fkpxb77b33fjn2

$ cp -a /gnu/store/nj79fxxl5wvnq7jpj2wgbx0591gkjw41-Python-3.5.9.tar.xz
Python-3.5.9.tar.xz
$ guix download Python-3.5.9.tar.xz
/gnu/store/9sa83nyjlm5dyhwys4imm1wa40mjaw1x-Python-3.5.9.tar.xz
0rkn451qfz3gbni57la00a5fbgish9jmm5bmhmgmf223vxwya447
----------------------------------------

Since the tarball is not restored to the original location the guix build
command still attempts the download and fails the offline build.

Greg Hogan

On Fri, Dec 11, 2020 at 9:42 PM Tobias Geerinckx-Rice <me@tobias.gr> wrote:

> Hullo Greg,
>
> Greg Hogan 写道:
> > If there is a better way to setup / configure / execute offline
> > builds
> > please let me know!
>
> ...yes :-)
>
> > I am attempting an offline build without success. I have a Guix
> > 1.2.0 node
> > with internet access on which I download sources with transitive
> > dependencies:
> >   $ guix build --sources=transitive tzdata > ~/transfer
>
> OK.
>
> > I then copy the files as root to a Guix 1.2.0 node without
> > internet access
> > (only local network access):
> >   # cat /home/<user>/transfer | xargs -n 1 -I{} scp -p {}
> >   <ip>:{}
>
> Now you've basically reinvented ‘guix copy --to=<ip>’, but in a
> way that won't update the store database in /var/guix/db.  I'm
> afraid that won't work.
>
> Guix won't ‘see’ the files you copy to the remote store and will
> consider them G to be C'd next time you run ‘guix gc’.  Or in this
> case:
>
> > Guix starts downloading and the transferred file is gone!
>
> Same thing.
>
> > I'm lost as to
> > why a new download attempt is made as the file data and
> > timestamps match
> > the original server.
>
> If the file isn't registered in the database, the store item is
> never considered valid.  Guix doesn't (yet) care about the
> data/timestamps at this point.
>
> If there's a reason you can't/won't use ‘guix copy’, you might
> work around that by copying each file in ~/transfer to, say,
> <ip>:/tmp/staging (instead of <ip>:/gnu/store), then running ‘guix
> download /tmp/staging/<file>...’ on the remote host.
>
> Kind regards,
>
> T G-R
>

[-- Attachment #2: Type: text/html, Size: 4266 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Offline build failure
  2020-12-14 19:33   ` Greg Hogan
@ 2020-12-15  2:47     ` Mark H Weaver
  0 siblings, 0 replies; 4+ messages in thread
From: Mark H Weaver @ 2020-12-15  2:47 UTC (permalink / raw)
  To: Greg Hogan, Tobias Geerinckx-Rice; +Cc: guix-devel

Hi Greg,

Greg Hogan <code@greghogan.com> writes:

> For the Python-3.5.9 tarball the first version loads in the expected way.
> The second tarball is restored to a new location:
>
> ----------------------------------------
> $ cp -a /gnu/store/f99fblkzb6ip268sg096shhs7wzjyp55-Python-3.5.9.tar.xz
> Python-3.5.9.tar.xz
> $ guix download Python-3.5.9.tar.xz
> /gnu/store/f99fblkzb6ip268sg096shhs7wzjyp55-Python-3.5.9.tar.xz
> 0jdh9pvx6m6lfz2liwvvhn7vks7qrysqgwn517fkpxb77b33fjn2
>
> $ cp -a /gnu/store/nj79fxxl5wvnq7jpj2wgbx0591gkjw41-Python-3.5.9.tar.xz
> Python-3.5.9.tar.xz
> $ guix download Python-3.5.9.tar.xz
> /gnu/store/9sa83nyjlm5dyhwys4imm1wa40mjaw1x-Python-3.5.9.tar.xz
> 0rkn451qfz3gbni57la00a5fbgish9jmm5bmhmgmf223vxwya447
> ----------------------------------------
>
> Since the tarball is not restored to the original location the guix build
> command still attempts the download and fails the offline build.

guix download will only work properly for the outputs of "fixed-output"
derivations, i.e. derivations for which Guix knows, in advance, the
sha256 hash of the resulting output.  This roughly corresponds to store
items that either downloaded or imported directly into the store from
local files (e.g. patches).

It will *not* work properly for the outputs of derivations that are
computed somehow, where Guix does *not* know in advance the hash of the
resulting output.  In particular, it won't work properly for outputs
that result from applying patches or snippets.

My guess is that the second tarball above was the result of applying
patches to the downloaded tarball.

To copy arbitrary store items from one machine to another, you need
"guix copy" or "guix archive".

      Regards,
        Mark


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2020-12-15  2:48 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-12-11 21:11 Offline build failure Greg Hogan
2020-12-11 21:42 ` Tobias Geerinckx-Rice
2020-12-14 19:33   ` Greg Hogan
2020-12-15  2:47     ` Mark H Weaver

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).