Using Guix archive (Help!)

Using Guix archive (Help!)

[Pjotr Prins]

Hi,

I am trying to send a NAR archive to Fred. We are facing a few issues.
First I created the archive

   env GUIX_PACKAGE_PATH=../guix-bioinformatics/ ./pre-inst-env guix archive --export -r --no-grafts genenetwork2 > gn2.nar

and Fred installed the key and unpacked it. Unfortunately it does not
contain package dependencies, such as mysql:

   https://gitlab.com/genenetwork/guix-bioinformatics/blob/master/gn/packages/genenetwork.scm#L194

somehow propagated-inputs don't go into the archive. You'd think they are critical
dependencies.

  Q1: is that to be expected? 

NAR files are signed. I think it would be very useful to be able to
distribute unsigned NAR files.

  Q2: can signing/checking be made optional?

I can't view the contents.

  Q3: how do we view the contents of a NAR file without unpacking it?

and

  Q4: is there a nice way to include a profile path in the store that
      can be used quickly?

The alternative is tarballing an environment/container, but the
problem there is that Fred's Guix daemon won't be aware of the new
packages. We want to avoid using Docker.

  Q5: is there an elegant way to add those unpacked store items to the
  database?

Thanks! I tried using NAR Files before and gave up. I hope it is clear
why.

Pj.

Re: Using Guix archive (Help!)

[Pjotr Prins]

I guess no one uses this format ;). I set up another guix publish
server for Fred (Africa could not reach guix.genenetwork.org for some
reason). 

I know we can use Guix pack but that don't play well with a running
Guix system. Now if we could unpack it would be solved.

I'll have a think about this.

Pj.

On Wed, Jan 03, 2018 at 12:08:36PM +0100, Pjotr Prins wrote:
> Hi,
> 
> I am trying to send a NAR archive to Fred. We are facing a few issues.
> First I created the archive
> 
>    env GUIX_PACKAGE_PATH=../guix-bioinformatics/ ./pre-inst-env guix archive --export -r --no-grafts genenetwork2 > gn2.nar
> 
> and Fred installed the key and unpacked it. Unfortunately it does not
> contain package dependencies, such as mysql:
> 
>    https://gitlab.com/genenetwork/guix-bioinformatics/blob/master/gn/packages/genenetwork.scm#L194
> 
> somehow propagated-inputs don't go into the archive. You'd think they are critical
> dependencies.
> 
>   Q1: is that to be expected? 
> 
> NAR files are signed. I think it would be very useful to be able to
> distribute unsigned NAR files.
> 
>   Q2: can signing/checking be made optional?
> 
> I can't view the contents.
> 
>   Q3: how do we view the contents of a NAR file without unpacking it?
> 
> and
> 
>   Q4: is there a nice way to include a profile path in the store that
>       can be used quickly?
> 
> The alternative is tarballing an environment/container, but the
> problem there is that Fred's Guix daemon won't be aware of the new
> packages. We want to avoid using Docker.
> 
>   Q5: is there an elegant way to add those unpacked store items to the
>   database?
> 
> Thanks! I tried using NAR Files before and gave up. I hope it is clear
> why.
> 
> Pj.
> 

-- 

Re: Using Guix archive (Help!)

[Ricardo Wurmus]

Hi Pjotr,

> somehow propagated-inputs don't go into the archive. You'd think they are critical
> dependencies.
>
>   Q1: is that to be expected?

I’m not sure, but it seems wrong.  What I used to do is to create a
profile and export that recursively:

  guix archive --export --recursive \
    $(readlink -f /project/.guix-profile) | \
    gzip --stdout - > my-profile.nar.gz

This should include propagated inputs.

> NAR files are signed. I think it would be very useful to be able to
> distribute unsigned NAR files.
>
>   Q2: can signing/checking be made optional?



> I can't view the contents.
>
>   Q3: how do we view the contents of a NAR file without unpacking it?

I don’t think you can.  It’s a custom format implemented by
nix/libutil/archive.cc.

>   Q4: is there a nice way to include a profile path in the store that
>       can be used quickly?

What do you mean by “include … in the store”?  With the above command to
recursively export profiles you get a single archive that can be
imported to the store as a profile.

>   Q5: is there an elegant way to add those unpacked store items to the
>   database?

What unpacked store items do you mean?

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Using Guix archive (Help!)

[Pjotr Prins]

On Sat, Jan 06, 2018 at 01:46:18PM +0100, Ricardo Wurmus wrote:
> I’m not sure, but it seems wrong.  What I used to do is to create a
> profile and export that recursively:
> 
>   guix archive --export --recursive \
>     $(readlink -f /project/.guix-profile) | \
>     gzip --stdout - > my-profile.nar.gz
> 
> This should include propagated inputs.

I'll try that again, but it does not appear to be so elegant. Besides
an archive is an archive ;). It should be a 'closure'.

> > NAR files are signed. I think it would be very useful to be able to
> > distribute unsigned NAR files.
> >
> >   Q2: can signing/checking be made optional?
> 
> 
> 
> > I can't view the contents.
> >
> >   Q3: how do we view the contents of a NAR file without unpacking it?
> 
> I don’t think you can.  It’s a custom format implemented by
> nix/libutil/archive.cc.

Yes, we ought to do something about that. Nix nar files could be be
unpacked before. I'll take a look at that. Not goot to have obscure
formats.

> >   Q4: is there a nice way to include a profile path in the store that
> >       can be used quickly?
> 
> What do you mean by “include … in the store”?  With the above command to
> recursively export profiles you get a single archive that can be
> imported to the store as a profile.

It is elegant (if it works) and not elegant at the same time. I am
thinking more of a symlink -S option like we have with pack.

> >   Q5: is there an elegant way to add those unpacked store items to the
> >   database?
> 
> What unpacked store items do you mean?

I mean how to add the contents of a pack image to the store. What
archive does, but then via the pack command.

Well, if archive works properly, pack does not need to do anything ;)

I know pack serves one purpose - deployment without guix - archive
another - deployment with guix. It is that archive is not working so
well, right now. pack could be supporting deployment on guix and
archive could behave a bit more like pack.

I'll take a look at the archive format.

Pj.

Re: Using Guix archive (Help!)

[Pjotr Prins]

> On Sat, Jan 06, 2018 at 01:46:18PM +0100, Ricardo Wurmus wrote:
> > >   Q3: how do we view the contents of a NAR file without unpacking it?
> > 
> > I don’t think you can.  It’s a custom format implemented by
> > nix/libutil/archive.cc.

OK, the format is pretty simple. On why Nix did not use tar instead:

  https://lethalman.blogspot.nl/2014/08/nix-pill-9-automatic-runtime.html

I kind of like the way nar files work(ed) in Nix. No signing, no
assumptions. Much like tar. If we can view nar files and just add
their contents to the store we would have another powerful
distribution mechanism. Especially in controlled environments. At this
point I have to set up two publish servers just to pass packages
along. I have to tell others how to add a key and substitute URLs.

It is overkill. Also nar files can be one-click installs with or
without a store.

And for the record, I am no fan of signed binaries installs. Much of
the world does not require you to install a key first. When a user
installs software we can assume he knows what he is doing and trusts
the source (rightfully or not). Signing is good, but it should be
optional.

Same for a default Guix binary install. By installing Guix we trust
the source implicitely. But I a know I am heretic screaming into the
void ;). I'll scream once a year, nevertheless. I know screaming is
easy, so maybe I can help a little here.

I have limited internet these weeks, but when I have more space I can
look at Nix tooling again. See if there is something to view the
contents of a nar file. If we want to stick to NAR we can improve the
use of them. That would give Guix nar, pack, Docker and my relocatable
binary installs. Choice is good.

Pj.

Re: Using Guix archive (Help!)

[Tobias Geerinckx-Rice]

Pjotr,

Pjotr Prins wrote on 07/01/18 at 11:32:
> I have limited internet these weeks, but when I have more space I can
> look at Nix tooling again. See if there is something to view the
> contents of a nar file.

Apparently[0] Nix 1.12 has a new ‘nix’ UI or wrapper with direct support
for listing NAR contents.

It[1] seems to work on modern-day (decompressed) Guix NARs too:

  $ curl https://mirror.hydra.gnu.org/guix/nar\
  > /gzip/5nhqwkvy7v51jd8lf4krljfbl01v6rx3-e2fsprogs-1.43.6 > my.nar.gz
  $ gunzip my.nar.gz
  $ /nix/store/*/bin/nix ls-nar -lR my.nar /
  dr-xr-xr-x                    0 ./bin
  -r-xr-xr-x                18216 ./bin/chattr
  -r-xr-xr-x                 1540 ./bin/compile_et
  -r-xr-xr-x                13888 ./bin/lsattr
  -r-xr-xr-x                 1356 ./bin/mk_cmds
  dr-xr-xr-x                    0 ./etc
  -r--r--r--                  808 ./etc/mke2fs.conf
  dr-xr-xr-x                    0 ./include
  lrwxrwxrwx                    0 ./include/com_err.h -> et/com_err.h
  dr-xr-xr-x                    0 ./include/e2p
  [...]

I haven't tried more than the above commands, nor do I use Nix anymore,
nor do I have any idea how good or reusable the code is, but it's a start.

Kind regards,

T G-R

[0]: https://github.com/NixOS/nix/issues/1257
[1]:
https://hydra.nixos.org/build/67181169/download/1/nix-1.12pre5839_6e098968-x86_64-linux.tar.bz2

Re: Using Guix archive (Help!)

[Pjotr Prins]

On Mon, Jan 08, 2018 at 04:15:52PM +0100, Tobias Geerinckx-Rice wrote:
> I haven't tried more than the above commands, nor do I use Nix anymore,
> nor do I have any idea how good or reusable the code is, but it's a start.

Absolutely! I'll work on this. Thanks,

Pj.

Re: Using Guix archive (Help!)

[Ludovic Courtès]

Hello!

Ricardo Wurmus <rekado@elephly.net> skribis:

>> somehow propagated-inputs don't go into the archive. You'd think they are critical
>> dependencies.
>>
>>   Q1: is that to be expected?
>
> I’m not sure, but it seems wrong.

I understand it’s weird, but it’s expected: ‘guix archive’ is a
low-level thing that works at the store level.  Propagated inputs are a
package-level concept, so it doesn’t know about it.

>> NAR files are signed. I think it would be very useful to be able to
>> distribute unsigned NAR files.
>>
>>   Q2: can signing/checking be made optional?

No, on purpose.

However unsigned archives would be useful for content-addressed items
(e.g., source code downloads), and it’s currently not supported.

>>   Q3: how do we view the contents of a NAR file without unpacking it?
>
> I don’t think you can.  It’s a custom format implemented by
> nix/libutil/archive.cc.

It’s also implemented in (guix nar).  :-)

So it Wouldn’t Be Hard to implement a content browsers and support “guix
archive -t”.

Ludo’.