‘guix publish’ now compresses archives

‘guix publish’ now compresses archives

[Ludovic Courtès]

Hi!

‘guix publish --compression’ can now compress archives in gzip format,
using zlib, which makes it much more usable in real life.  See commit
4a1fc562ae5eedf40f6ae4eabe30580b0983b8f6.

‘guix publish’ is cache-friendly: it uses /nar/gzip URLs for gzipped
archives, and /nar for uncompressed archives.  That way, even if ‘guix
publish’ is restarted with different compression parameters, previously
announced /nar/* URLs remain valid.

Please try it and report back.

To recap, the difficulty with all this is that we couldn’t just call out
the ‘gzip’ command because ‘guix publish’ uses threads, and threads and
fork(2) don’t go together well.  So we needed zlib bindings, and when
discussing it earlier with Dave, I thought we’d have to wrap zlib’s
low-level API, which is painful, so we were sad and all.  Next we had
this hydra.gnu.org outage, which entailed more sadness.

Then I realized that zlib has this high-level ‘gz’ API, which is easy
and does what we need; hence, (guix zlib) provides bindings to that.
The only limitation is that it needs a file descriptor as its source or
sink and cannot compress into memory.  That’s enough for ‘guix publish’
though.

Comments welcome!

Ludo’.

Re: ‘guix publish’ now compresses archives

[Tomáš Čech]

[-- Attachment #1: Type: text/plain, Size: 2527 bytes --]

On Tue, Jul 19, 2016 at 12:22:42AM +0200, Ludovic Courtès wrote:
>Hi!
>
>‘guix publish --compression’ can now compress archives in gzip format,
>using zlib, which makes it much more usable in real life.  See commit
>4a1fc562ae5eedf40f6ae4eabe30580b0983b8f6.
>
>‘guix publish’ is cache-friendly: it uses /nar/gzip URLs for gzipped
>archives, and /nar for uncompressed archives.  That way, even if ‘guix
>publish’ is restarted with different compression parameters, previously
>announced /nar/* URLs remain valid.
>
>Please try it and report back.
>
>To recap, the difficulty with all this is that we couldn’t just call out
>the ‘gzip’ command because ‘guix publish’ uses threads, and threads and
>fork(2) don’t go together well.  So we needed zlib bindings, and when
>discussing it earlier with Dave, I thought we’d have to wrap zlib’s
>low-level API, which is painful, so we were sad and all.  Next we had
>this hydra.gnu.org outage, which entailed more sadness.
>
>Then I realized that zlib has this high-level ‘gz’ API, which is easy
>and does what we need; hence, (guix zlib) provides bindings to that.
>The only limitation is that it needs a file descriptor as its source or
>sink and cannot compress into memory.  That’s enough for ‘guix publish’
>though.
>
>Comments welcome!

When I saw archive improvement I remembered I have a comment about
that (but not associated to compression).

Recently when Snappy and Flatpak news flew around the world I saw some
people were thinking about Guix as possible and cleaner alternative to
new package types.

As I am happy to see enthusiasm about Guix, I don't think that current
archive format is usable for distribution of binary packages of
whatever origin. Problem is that we have package definition as part of
package manager GIT snapshot or local files but this information is
not distributed along with the package archive.

If this area is interesting for us, we can do better. It would be
great to be able to verify such archive, maintain the reproducibility
or provide recipe how to bake missing dependencies.

I was thinking that we could (and should) add some optional support
for metadata which could describe the origin of the package. It could
contain:
 - Guix GIT revision used
 - differences against GIT revision
 - local package recipe
 - exported dependency tree of package recipes
 - optionally URL where missing dependencies can be found

What do you think?

S_W

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: ‘guix publish’ now compresses archives

[Pjotr Prins]

On Tue, Jul 19, 2016 at 08:29:15AM +0200, Tomáš Čech wrote:
> I was thinking that we could (and should) add some optional support
> for metadata which could describe the origin of the package. It could
> contain:
> - Guix GIT revision used
> - differences against GIT revision
> - local package recipe
> - exported dependency tree of package recipes
> - optionally URL where missing dependencies can be found

The package would need to be signed and contain all its dependencies.
This is what guix archive does when you add some metadata.

The size of the package does not matter (much). If Docker can be
popular...

I agree it would be an idea to decouple from publish daemons in the
form of downloadable files. Nix had this feature - not sure it is still
there.

Pj.

Re: ‘guix publish’ now compresses archives

[Ludovic Courtès]

Hello Tomáš,

Tomáš Čech <sleep_walker@gnu.org> skribis:

> As I am happy to see enthusiasm about Guix, I don't think that current
> archive format is usable for distribution of binary packages of
> whatever origin. Problem is that we have package definition as part of
> package manager GIT snapshot or local files but this information is
> not distributed along with the package archive.

I’m not sure what you mean.  The “substitute” mechanism is meant as a
cache.  Guix locally compiles the package recipe to a low-level
representation of the build process (a derivation), and then, it can
*optionally* ask substitute servers whether they have the result
corresponding to that derivation.

A derivation is self-contained and unambiguous.  I can do:

--8<---------------cut here---------------start------------->8---
$ guix build -d sed
/gnu/store/whkb4d5sq2326m54zpdz6hhmi37jy6j4-sed-4.2.2.drv
$ cat $(guix build -d sed) | cut -d ']' -f 1
Derive([("out","/gnu/store/4ls9kdy1w6ichvmbrl5wn98lxmznbkd6-sed-4.2.2","","")
$ wget -O /tmp/sed-binary.nar.bz2 https://mirror.hydra.gnu.org/nar/4ls9kdy1w6ichvmbrl5wn98lxmznbkd6-sed-4.2.2
--2016-07-19 15:09:02--  https://mirror.hydra.gnu.org/nar/4ls9kdy1w6ichvmbrl5wn98lxmznbkd6-sed-4.2.2
Ni solvigas mirror.hydra.gnu.org (mirror.hydra.gnu.org)... 131.159.14.26, 2001:4ca0:2001:10:225:90ff:fedb:c720
Konektado al mirror.hydra.gnu.org (mirror.hydra.gnu.org)|131.159.14.26|:443... konektita.
HTTP peto sendita, ni atendas respondon... 200 OK
Grando: nespecifita [application/x-nix-archive]
Ni konservas al: '/tmp/sed-binary.nar.bz2'

/tmp/sed-binary.nar     [ <=>                ] 145.07K  --.-KB/s    in 0.07s   

2016-07-19 15:09:02 (2.04 MB/s) - '/tmp/sed-binary.nar.bz2' konservita [148549]
--8<---------------cut here---------------end--------------->8---

… and now I have the binary corresponds to that sed-4.2.2.drv, or so
claims mirror.hydra.gnu.org.  (This URL scheme is what ‘guix publish’
implements.)

Thus, mirror.hydra.gnu.org does not need to know about the (package …)
form, the Git revision of Guix, whether the recipe was local or not,
etc.

Of course I have to trust mirror.hydra.gnu.org to genuinely serve
binaries resulting corresponding to the derivations.  To express this
trust, I register its archive signing key, as explained at
<https://www.gnu.org/software/guix/manual/html_node/Substitutes.html>.

Does that clarify things, or am I misunderstanding?

Thanks,
Ludo’.

Re: ‘guix publish’ now compresses archives

[Tomáš Čech]

[-- Attachment #1: Type: text/plain, Size: 3515 bytes --]

On Tue, Jul 19, 2016 at 03:15:38PM +0200, Ludovic Courtès wrote:
>Hello Tomáš,
>
>Tomáš Čech <sleep_walker@gnu.org> skribis:
>
>> As I am happy to see enthusiasm about Guix, I don't think that current
>> archive format is usable for distribution of binary packages of
>> whatever origin. Problem is that we have package definition as part of
>> package manager GIT snapshot or local files but this information is
>> not distributed along with the package archive.
>
>I’m not sure what you mean.  The “substitute” mechanism is meant as a
>cache.  Guix locally compiles the package recipe to a low-level
>representation of the build process (a derivation), and then, it can
>*optionally* ask substitute servers whether they have the result
>corresponding to that derivation.
>
>A derivation is self-contained and unambiguous.  I can do:
>
>--8<---------------cut here---------------start------------->8---
>$ guix build -d sed
>/gnu/store/whkb4d5sq2326m54zpdz6hhmi37jy6j4-sed-4.2.2.drv
>$ cat $(guix build -d sed) | cut -d ']' -f 1
>Derive([("out","/gnu/store/4ls9kdy1w6ichvmbrl5wn98lxmznbkd6-sed-4.2.2","","")
>$ wget -O /tmp/sed-binary.nar.bz2 https://mirror.hydra.gnu.org/nar/4ls9kdy1w6ichvmbrl5wn98lxmznbkd6-sed-4.2.2
>--2016-07-19 15:09:02--  https://mirror.hydra.gnu.org/nar/4ls9kdy1w6ichvmbrl5wn98lxmznbkd6-sed-4.2.2
>Ni solvigas mirror.hydra.gnu.org (mirror.hydra.gnu.org)... 131.159.14.26, 2001:4ca0:2001:10:225:90ff:fedb:c720
>Konektado al mirror.hydra.gnu.org (mirror.hydra.gnu.org)|131.159.14.26|:443... konektita.
>HTTP peto sendita, ni atendas respondon... 200 OK
>Grando: nespecifita [application/x-nix-archive]
>Ni konservas al: '/tmp/sed-binary.nar.bz2'
>
>/tmp/sed-binary.nar     [ <=>                ] 145.07K  --.-KB/s    in 0.07s
>
>2016-07-19 15:09:02 (2.04 MB/s) - '/tmp/sed-binary.nar.bz2' konservita [148549]
>--8<---------------cut here---------------end--------------->8---
>
>… and now I have the binary corresponds to that sed-4.2.2.drv, or so
>claims mirror.hydra.gnu.org.  (This URL scheme is what ‘guix publish’
>implements.)
>
>Thus, mirror.hydra.gnu.org does not need to know about the (package …)
>form, the Git revision of Guix, whether the recipe was local or not,
>etc.
>
>Of course I have to trust mirror.hydra.gnu.org to genuinely serve
>binaries resulting corresponding to the derivations.  To express this
>trust, I register its archive signing key, as explained at
><https://www.gnu.org/software/guix/manual/html_node/Substitutes.html>.
>
>Does that clarify things, or am I misunderstanding?

Thanks for lenghty mail, but I'm sorry, no, I mean something
different. Let's try example situation instead.

Imagine situation where person A is running some distribution with
Guix package manager on top and has some set of his personal packages
containing additional patches which are not part of Guix GIT.

He'd like to share the package with person B, running different
distribution with Guix package manager at different revision on top,
with different set of personal packages and alterations.

I'd like to provide them a way, how to pass from person A to person B
some binary archive in a way that he could understand (and verify)
what he received. If it requries out-of-tree package definitions of
person A, patches, etc, bundle it together.

IOW, prepare Guix for being package manager for packages which are not
defined in Guix GIT.

Is that better understood?

Thanks,

S_W

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: ‘guix publish’ now compresses archives

[Ricardo Wurmus]

Tomáš Čech <sleep_walker@gnu.org> writes:

> Imagine situation where person A is running some distribution with
> Guix package manager on top and has some set of his personal packages
> containing additional patches which are not part of Guix GIT.
>
> He'd like to share the package with person B, running different
> distribution with Guix package manager at different revision on top,
> with different set of personal packages and alterations.
>
> I'd like to provide them a way, how to pass from person A to person B
> some binary archive in a way that he could understand (and verify)
> what he received. If it requries out-of-tree package definitions of
> person A, patches, etc, bundle it together.

This is already possible with “guix archive --export” and “--import”.

For the users at the MDC I wrote a little guide on how to share software
environments with other people so that others can reproduce the exact
same state.  The first way to share an environment is symbolic,
i.e. a manifest + Guix git hash + git hashes of any additional package
repositories.

The second way to share an environment is the binary method.  With this
command you can export a complete profile (e.g. at
“/project/.guix-profile”) and write it to a gzipped archive:

#+BEGIN_SRC
guix archive --export --recursive \
  $(readlink -f /project/.guix-profile) | \
  gzip --stdout - > my-profile.nar.gz
#+END_SRC

The recipient can take this archive and import it into their store after
authenticating the public key of the sender.  The result of importing is
that a bunch of store items appear in the receiver’s Guix store.

Note that the package artifacts in the exported profile don’t have to be
the result of building packages from the package expressions in upstream
Guix.  They can include packages that were only defined in some
directory of GUIX_PACKAGE_PATH or that have been imported from another
store.

What’s not so nice about this is that you can end up with binaries in
your store that you cannot rebuild yourself (because you never had the
sources to begin with).  (I wonder if this has implications on software
freedom.)

~~ Ricardo

Re: ‘guix publish’ now compresses archives

[Tomáš Čech]

[-- Attachment #1: Type: text/plain, Size: 1538 bytes --]

On Tue, Jul 19, 2016 at 04:23:14PM +0200, Ricardo Wurmus wrote:
>
>Tomáš Čech <sleep_walker@gnu.org> writes:
>
>> Imagine situation where person A is running some distribution with
>> Guix package manager on top and has some set of his personal packages
>> containing additional patches which are not part of Guix GIT.
>>
>> He'd like to share the package with person B, running different
>> distribution with Guix package manager at different revision on top,
>> with different set of personal packages and alterations.
>>
>> I'd like to provide them a way, how to pass from person A to person B
>> some binary archive in a way that he could understand (and verify)
>> what he received. If it requries out-of-tree package definitions of
>> person A, patches, etc, bundle it together.
>
>This is already possible with “guix archive --export” and “--import”.
> ...
>What’s not so nice about this is that you can end up with binaries in
>your store that you cannot rebuild yourself (because you never had the
>sources to begin with).  (I wonder if this has implications on software
>freedom.)
>
>~~ Ricardo
>

Ooops, I answered to Ricardo only...

Yes, you can take exported archive (optionally with it's dependencies
- --recursive) and add it to the person B's store. That is comparable
to tarball (and making Guix aware of new items in store).

The thing that you can't rebuild the binaries is exactly my point,
because person A knows how to build it, but person B does not. Thanks!

S_W

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: ‘guix publish’ now compresses archives

[Ricardo Wurmus]

Tomáš Čech <tcech@suse.com> writes:

> On Tue, Jul 19, 2016 at 04:23:14PM +0200, Ricardo Wurmus wrote:
>>
>>Tomáš Čech <sleep_walker@gnu.org> writes:
>>
>>> Imagine situation where person A is running some distribution with
>>> Guix package manager on top and has some set of his personal packages
>>> containing additional patches which are not part of Guix GIT.
>>>
>>> He'd like to share the package with person B, running different
>>> distribution with Guix package manager at different revision on top,
>>> with different set of personal packages and alterations.
>>>
>>> I'd like to provide them a way, how to pass from person A to person B
>>> some binary archive in a way that he could understand (and verify)
>>> what he received. If it requries out-of-tree package definitions of
>>> person A, patches, etc, bundle it together.
>>
>>This is already possible with “guix archive --export” and “--import”.
>> ...
>>What’s not so nice about this is that you can end up with binaries in
>>your store that you cannot rebuild yourself (because you never had the
>>sources to begin with).  (I wonder if this has implications on software
>>freedom.)
>>
>>~~ Ricardo
>>
>
> Ooops, I answered to Ricardo only...
>
> Yes, you can take exported archive (optionally with it's dependencies
> - --recursive) and add it to the person B's store. That is comparable
> to tarball (and making Guix aware of new items in store).
>
> The thing that you can't rebuild the binaries is exactly my point,
> because person A knows how to build it, but person B does not. Thanks!

Are you suggesting that package archives and store exports must include
all package expressions (and all supporting code to build them) needed
to reproduce them?  The store can hold packages that were built with
different versions of Guix.  Does this mean that each version of Guix
that has been used should also be kept in the store?

But wait, this goes even further than just versions of Guix.  There may
be stuff in the GUIX_PACKAGE_PATH.  Are you suggesting that all of this
should become part of package archives?

If so: how?

~~ Ricardo

Re: ‘guix publish’ now compresses archives

[Tomáš Čech]

[-- Attachment #1: Type: text/plain, Size: 3671 bytes --]

On Wed, Jul 20, 2016 at 01:20:26PM +0200, Ricardo Wurmus wrote:
>
>Tomáš Čech <tcech@suse.com> writes:
>
>> On Tue, Jul 19, 2016 at 04:23:14PM +0200, Ricardo Wurmus wrote:
>>>
>>>Tomáš Čech <sleep_walker@gnu.org> writes:
>>>
>>>> Imagine situation where person A is running some distribution with
>>>> Guix package manager on top and has some set of his personal packages
>>>> containing additional patches which are not part of Guix GIT.
>>>>
>>>> He'd like to share the package with person B, running different
>>>> distribution with Guix package manager at different revision on top,
>>>> with different set of personal packages and alterations.
>>>>
>>>> I'd like to provide them a way, how to pass from person A to person B
>>>> some binary archive in a way that he could understand (and verify)
>>>> what he received. If it requries out-of-tree package definitions of
>>>> person A, patches, etc, bundle it together.
>>>
>>>This is already possible with “guix archive --export” and “--import”.
>>> ...
>>>What’s not so nice about this is that you can end up with binaries in
>>>your store that you cannot rebuild yourself (because you never had the
>>>sources to begin with).  (I wonder if this has implications on software
>>>freedom.)
>>>
>>>~~ Ricardo
>>>
>>
>> Ooops, I answered to Ricardo only...
>>
>> Yes, you can take exported archive (optionally with it's dependencies
>> - --recursive) and add it to the person B's store. That is comparable
>> to tarball (and making Guix aware of new items in store).
>>
>> The thing that you can't rebuild the binaries is exactly my point,
>> because person A knows how to build it, but person B does not. Thanks!
>
>Are you suggesting that package archives and store exports must include
>all package expressions (and all supporting code to build them) needed
>to reproduce them?  The store can hold packages that were built with
>different versions of Guix.  Does this mean that each version of Guix
>that has been used should also be kept in the store?

First, I'm not saying that we should do that for every archive, but I
think that having a way how to automatically export this information
would be great and I see it as a week point for using Guix packages as
alternative to Snappy or Flatpak.

Now, as long as Guix sources are part of equation and change during
the time, you may need to keep more Guix versions. If there is some
stable not-changing core, you probably could eliminate the need.

>But wait, this goes even further than just versions of Guix.  There may
>be stuff in the GUIX_PACKAGE_PATH.  Are you suggesting that all of this
>should become part of package archives?

This doesn't sound like tougher problem than the rest - you have
separate package definitions. It means just more data to process.
(that doesn't mean that problem is easy to solve)

>If so: how?

1] I guess I should better explain my position. In the discussion on
social media I was explaining why it can't be used now for
self-containing packages and what is missing. My goal now on the
mailing list is here the discussion itself, to see whether it is
possible and whether are people interested in that.

2] I don't know much Guile, maybe there is nice way how to work with
Scheme program from Scheme program. If I should implement it, I'd need
to strictly "draw line" between package definitions and Guix itself,
then I would need to parse package definitions files to identify which
parts were used and ship those. Maybe you can get this information
during evaluation by some side effect. I don't know Guile enough, really.

S_W

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: ‘guix publish’ now compresses archives

[Thompson, David]

On Wed, Jul 20, 2016 at 9:05 AM, Tomáš Čech <sleep_walker@gnu.org> wrote:

> First, I'm not saying that we should do that for every archive, but I
> think that having a way how to automatically export this information
> would be great and I see it as a week point for using Guix packages as
> alternative to Snappy or Flatpak.

I don't really understand the point of this back-and-forth.  It's
quite simple: If the user builds the same package expression with the
same version of Guix, they will get the same result if the build is
deterministic.  I don't understand the contrast with Snappy and
Flatpak because they don't provide this feature at all, opting instead
to provide opaque binaries with no real provenance.  I can only assume
that there is some fundamental misunderstanding about Guix going on
here.

- Dave

Re: ‘guix publish’ now compresses archives

[Tomáš Čech]

[-- Attachment #1: Type: text/plain, Size: 1200 bytes --]

On Wed, Jul 20, 2016 at 09:12:53AM -0400, Thompson, David wrote:
>On Wed, Jul 20, 2016 at 9:05 AM, Tomáš Čech <sleep_walker@gnu.org> wrote:
>
>> First, I'm not saying that we should do that for every archive, but I
>> think that having a way how to automatically export this information
>> would be great and I see it as a week point for using Guix packages as
>> alternative to Snappy or Flatpak.
>
>I don't really understand the point of this back-and-forth.  It's
>quite simple: If the user builds the same package expression with the
>same version of Guix, they will get the same result if the build is
>deterministic.

Yes.

>I don't understand the contrast with Snappy and Flatpak because they
>don't provide this feature at all, opting instead to provide opaque
>binaries with no real provenance.

Snappy and Flatpak does not provide this feature but Guix could.

>I can only assume that there is some fundamental misunderstanding
>about Guix going on here.

I don't think so. My guess is that you're using Guix too much as OS
and too little as package manager in another OS.

Nevermind, enough of this. It seems that noone is interested in that
anyway.

S_W

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: ‘guix publish’ now compresses archives

[Ricardo Wurmus]

Thompson, David <dthompson2@worcester.edu> writes:

> On Wed, Jul 20, 2016 at 9:05 AM, Tomáš Čech <sleep_walker@gnu.org> wrote:
>
>> First, I'm not saying that we should do that for every archive, but I
>> think that having a way how to automatically export this information
>> would be great and I see it as a week point for using Guix packages as
>> alternative to Snappy or Flatpak.
>
> I don't really understand the point of this back-and-forth.  It's
> quite simple: If the user builds the same package expression with the
> same version of Guix, they will get the same result if the build is
> deterministic.  I don't understand the contrast with Snappy and
> Flatpak because they don't provide this feature at all, opting instead
> to provide opaque binaries with no real provenance.  I can only assume
> that there is some fundamental misunderstanding about Guix going on
> here.

The point is that exporting a store item (or a package closure) is the
moral equivalent to producing an opaque binary.  The claim is that Guix
could do better here.  I agree to the first part but I’m not sure about
the second part.  It would be very nice if Guix really *could* do better
here without having to embed a copy of itself to each exported package.

One way to do that is to provide the equivalent of a source package,
i.e. exporting not only the closure but everything that’s needed to
build it yourself (including the source archives of all involved
packages and all package expressions and all of the supporting Guix
library features).

The weaker proposition is to embed some metadata (such as the exact
version of Guix upstream and a list of packages, but this would not be
sufficient as Guix can be modified and GUIX_PACKAGE_PATH would not be
included).

~~ Ricardo

Re: ‘guix publish’ now compresses archives

[Ricardo Wurmus]

Tomáš Čech <sleep_walker@gnu.org> writes:

> First, I'm not saying that we should do that for every archive, but I
> think that having a way how to automatically export this information
> would be great and I see it as a week point for using Guix packages as
> alternative to Snappy or Flatpak.

Agreed.  I think it’s unfortunate that there is no step in between the
purely symbolic and the purely binary method of sharing environments.

> Now, as long as Guix sources are part of equation and change during
> the time, you may need to keep more Guix versions. If there is some
> stable not-changing core, you probably could eliminate the need.

This is going to be virtually impossible.  Users can change Guix
locally.  We cannot assume an immutable core.

Guix packages are code.  Guix itself is mostly a library.

> 1] I guess I should better explain my position. In the discussion on
> social media I was explaining why it can't be used now for
> self-containing packages and what is missing. My goal now on the
> mailing list is here the discussion itself, to see whether it is
> possible and whether are people interested in that.

It *can* be used for self-contained packages.  “guix archive --export”
and “guix archive --import” is sufficient for people who come from the
world of bundling.  They can just take a binary archive that someone
else exported from their store and import it into their store.

It is not pretty and I wouldn’t do it, though, because I would always
want to be able to build the exact same binaries myself, which I cannot
do without the exact same version of Guix (this includes package
expressions).

> 2] I don't know much Guile, maybe there is nice way how to work with
> Scheme program from Scheme program. If I should implement it, I'd need
> to strictly "draw line" between package definitions and Guix itself,
> then I would need to parse package definitions files to identify which
> parts were used and ship those. Maybe you can get this information
> during evaluation by some side effect. I don't know Guile enough, really.

In Guix we don’t parse package definitions.  Package definitions are
code.  A package expression is evaluated to a Scheme value.  To compute
the Scheme value one needs all the definitions that are used therein.
And that’s the rest of Guix.  Package expressions are a part of the code
of Guix.  The code used inside of package expressions may change as we
encounter new problems that need to be addressed.  You cannot treat
package expressions as if they were apart from the rest of Guix.

If all you wanted was the names of all packages in a closure — that’s
easy to obtain with Guix.  Just recurse over the inputs.

But to get just a subset of Guix that is used in a closure is futile.
At the end all closures are rooted in bootstrap binaries, so it’s likely
you need a big chunk of all the Guix library definitions.  It’s easier
to just include a copy of Guix itself.

My point is that you cannot (currently?) do this after the fact for any
store item you may want to export.  You would have to embed the state of
the Guix world at store-add time in the store items themselves.  This
can be done by adding the current state of Guix to the store itself and
add it as an input to each store item.  This seems very heavy-handed,
though.

If we agree that we don’t need to be able to extract the Guix state for
*any* store item the problem is smaller and much easier to solve.
Preparing a reproducible binary package would then only involve
appending the current Guix state (including whatever Guix sees via
GUIX_PACKAGE_PATH) as a source archive to the exported closure.

~~ Ricardo

Re: ‘guix publish’ now compresses archives

[Andy Wingo]

On Tue 19 Jul 2016 00:22, ludo@gnu.org (Ludovic Courtès) writes:

> To recap, the difficulty with all this is that we couldn’t just call out
> the ‘gzip’ command because ‘guix publish’ uses threads, and threads and
> fork(2) don’t go together well.

FWIW open-pipe or open-process from (ice-9 popen) should work reliably.
The child is carefully written to be async-signal-safe before the
execve().

Andy

Re: ‘guix publish’ now compresses archives

[Ludovic Courtès]

Andy Wingo <wingo@igalia.com> skribis:

> On Tue 19 Jul 2016 00:22, ludo@gnu.org (Ludovic Courtès) writes:
>
>> To recap, the difficulty with all this is that we couldn’t just call out
>> the ‘gzip’ command because ‘guix publish’ uses threads, and threads and
>> fork(2) don’t go together well.
>
> FWIW open-pipe or open-process from (ice-9 popen) should work reliably.
> The child is carefully written to be async-signal-safe before the
> execve().

Yes, but ‘filtered-output-port’, used by ‘compressed-output-port’ in
(guix utils) is not implemented in terms of (ice-9 popen).

Ludo’.

Re: ‘guix publish’ now compresses archives

[Thompson, David]

On Thu, Jul 21, 2016 at 1:33 AM, Ricardo Wurmus
<ricardo.wurmus@mdc-berlin.de> wrote:
>
> Thompson, David <dthompson2@worcester.edu> writes:
>
>> On Wed, Jul 20, 2016 at 9:05 AM, Tomáš Čech <sleep_walker@gnu.org> wrote:
>>
>>> First, I'm not saying that we should do that for every archive, but I
>>> think that having a way how to automatically export this information
>>> would be great and I see it as a week point for using Guix packages as
>>> alternative to Snappy or Flatpak.
>>
>> I don't really understand the point of this back-and-forth.  It's
>> quite simple: If the user builds the same package expression with the
>> same version of Guix, they will get the same result if the build is
>> deterministic.  I don't understand the contrast with Snappy and
>> Flatpak because they don't provide this feature at all, opting instead
>> to provide opaque binaries with no real provenance.  I can only assume
>> that there is some fundamental misunderstanding about Guix going on
>> here.
>
> The point is that exporting a store item (or a package closure) is the
> moral equivalent to producing an opaque binary.  The claim is that Guix
> could do better here.  I agree to the first part but I’m not sure about
> the second part.  It would be very nice if Guix really *could* do better
> here without having to embed a copy of itself to each exported package.

Derivations are purposely a one-way street.  There's *no* way to get
from the derivation back to the source.  You always want to go from
the source to the derivation.  I think we're asking the wrong question
here.  It's not "I have a binary, now where is the source?", it's "I
have the source, now is there a binary available?"

- Dave

Re: ‘guix publish’ now compresses archives

[Tomáš Čech]

[-- Attachment #1: Type: text/plain, Size: 5207 bytes --]

On Thu, Jul 21, 2016 at 07:53:45AM +0200, Ricardo Wurmus wrote:
>
>Tomáš Čech <sleep_walker@gnu.org> writes:
>
>> First, I'm not saying that we should do that for every archive, but I
>> think that having a way how to automatically export this information
>> would be great and I see it as a week point for using Guix packages as
>> alternative to Snappy or Flatpak.
>
>Agreed.  I think it’s unfortunate that there is no step in between the
>purely symbolic and the purely binary method of sharing environments.
>
>> Now, as long as Guix sources are part of equation and change during
>> the time, you may need to keep more Guix versions. If there is some
>> stable not-changing core, you probably could eliminate the need.
>
>This is going to be virtually impossible.  Users can change Guix
>locally.  We cannot assume an immutable core.

Actually the condition can be this weak - there won't be any false
positives, it will only make preparing reproducible builds damn hard.

>Guix packages are code.  Guix itself is mostly a library.

That again supports the point that the code closure could gather
package code only when the library will be stable and feature complete
(unchanging).

>> 1] I guess I should better explain my position. In the discussion on
>> social media I was explaining why it can't be used now for
>> self-containing packages and what is missing. My goal now on the
>> mailing list is here the discussion itself, to see whether it is
>> possible and whether are people interested in that.
>
>It *can* be used for self-contained packages.  “guix archive --export”
>and “guix archive --import” is sufficient for people who come from the
>world of bundling.  They can just take a binary archive that someone
>else exported from their store and import it into their store.

OK, you are right.

>It is not pretty and I wouldn’t do it, though, because I would always
>want to be able to build the exact same binaries myself, which I cannot
>do without the exact same version of Guix (this includes package
>expressions).
>
>> 2] I don't know much Guile, maybe there is nice way how to work with
>> Scheme program from Scheme program. If I should implement it, I'd need
>> to strictly "draw line" between package definitions and Guix itself,
>> then I would need to parse package definitions files to identify which
>> parts were used and ship those. Maybe you can get this information
>> during evaluation by some side effect. I don't know Guile enough, really.
>
>In Guix we don’t parse package definitions.  Package definitions are
>code.  A package expression is evaluated to a Scheme value.  To compute
>the Scheme value one needs all the definitions that are used therein.

I'm aware of this. I'm not sure how lazy is evaluation in Guile but
interpreter knows every pieces of this information during the
execution.

If the interpreter can't share this information (and there are reasons
why not extend the language allowing querying the interpreter state
during the execution), it would need to be done with parser, that was
what I meant before (and yes, that wouldn't be elegant at all).

>And that’s the rest of Guix.  Package expressions are a part of the code
>of Guix.  The code used inside of package expressions may change as we
>encounter new problems that need to be addressed.  You cannot treat
>package expressions as if they were apart from the rest of Guix.

I think I have to just accept that.

>If all you wanted was the names of all packages in a closure — that’s
>easy to obtain with Guix.  Just recurse over the inputs.

That is apparently not enough.

>But to get just a subset of Guix that is used in a closure is futile.
>At the end all closures are rooted in bootstrap binaries, so it’s likely
>you need a big chunk of all the Guix library definitions.  It’s easier
>to just include a copy of Guix itself.

Right. And in the end it is not that bad solution either (until you
need to use two versions of Guix at the same time).

>My point is that you cannot (currently?) do this after the fact for any
>store item you may want to export.  You would have to embed the state of
>the Guix world at store-add time in the store items themselves.  This
>can be done by adding the current state of Guix to the store itself and
>add it as an input to each store item.  This seems very heavy-handed,
>though.

Right.

>If we agree that we don’t need to be able to extract the Guix state for
>*any* store item the problem is smaller and much easier to solve.
>Preparing a reproducible binary package would then only involve
>appending the current Guix state (including whatever Guix sees via
>GUIX_PACKAGE_PATH) as a source archive to the exported closure.

Yes, that sounds sufficiently complete and corresponds to intention -
not do that automatically, not to do that for all stores, but be able
to export information that when requested.

I'm afraid that in that case we'll never be able to easily combine two
or more exported closures together.

Thank you for your patience and input into this discussion.

Best regards,

S_W

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]