guix pack file enumerator?

guix pack file enumerator?

[Ricardo Wurmus]

Hi Guix,

“guix pack” is great for deployment of applications to servers that
don’t have Guix.  For a project I have a “deploy” target in my Makefile
that essentially does this:

    cat $(shell guix pack -RR -e '(load "guix.scm")' -S /bin=bin) | ssh remote-server "tar xvzf - -C /where/i/want/it"

This is fine for small deployments, but it’s a little annoying that it
transfer *all* the files, even those that haven’t changed.  So I thought
I could use rsync here, but it’s inconvenient that “guix pack” will do
what it was designed for and produce a single file bundle.

What do you think about adding an output format that is no format at all
but a file enumeration printed to stdout?  That way I could use “guix
pack” to produce a list of files to transfer and use that to transfer
only the unchanged files.  Alternatively, perhaps we could have a
“directory” format that merely copies (or links) the files to a new
directory root.

-- 
Ricardo

Re: guix pack file enumerator?

[Ryan Prior]

[-- Attachment #1: Type: text/plain, Size: 981 bytes --]

On December 6, 2020, Ricardo Wurmus <rekado@elephly.net> wrote:
> What do you think about adding an output format that is no format at
> all
> but a file enumeration printed to stdout? That way I could use “guix
> pack” to produce a list of files to transfer and use that to transfer
> only the unchanged files. Alternatively, perhaps we could have a
> “directory” format that merely copies (or links) the files to a new
> directory root.

These suggestions comport nicely with the Unix-philosophy and something
like that should probably be implemented for people who want to build
their own deploy systems like you're describing.

Additionally, I think that the use-case of "deploy this software to some
server that doesn't have Guix installed" is important enough that we
might want to support it explicitly such that "guix deploy" does the
right thing, doing minimal work to bring the target deployment up-to-
date. Is that feasible?

[-- Attachment #2: Type: text/html, Size: 3293 bytes --]

Re: guix pack file enumerator?

[zimoun]

Hi Ricardo,

On Sun, 06 Dec 2020 at 10:53, Ricardo Wurmus <rekado@elephly.net> wrote:

> What do you think about adding an output format that is no format at all
> but a file enumeration printed to stdout?  That way I could use “guix
> pack” to produce a list of files to transfer and use that to transfer
> only the unchanged files.  Alternatively, perhaps we could have a
> “directory” format that merely copies (or links) the files to a new
> directory root.

Well, I am not sure if it fits your use case but I am still annoyed to
find the “profile/manifest“ path when I use:

   guix pack -f docker --save-provenance -m manifest.scm

then extracts this “profile/manifest” needs:

--8<---------------cut here---------------start------------->8---
   docker export -o foo.tar <ID>

   tar -xf foo.tar $(tar -tf foo.tar | grep ’profile/manifest’)
--8<---------------cut here---------------end--------------->8---

in order to rebuild an approximation of ’manifest.scm+channels.scm’ if it
has been lost, for example to be able to re-run at anytime:

--8<---------------cut here---------------start------------->8---
   docker load < $(guix time-machine -C channels.scm \
                     -- pack -f docker \
                        -m manifest.scm)
   docker run -ti <TAG>
--8<---------------cut here---------------end--------------->8---

For reference, see <https://yhetil.org/guix/86sgb9nt4k.fsf@gmail.com>.


Therefore, I do not know if this output format is helping for this.


Thanks,
simon

Re: guix pack file enumerator?

[Ludovic Courtès]

Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

> “guix pack” is great for deployment of applications to servers that
> don’t have Guix.  For a project I have a “deploy” target in my Makefile
> that essentially does this:
>
>     cat $(shell guix pack -RR -e '(load "guix.scm")' -S /bin=bin) | ssh remote-server "tar xvzf - -C /where/i/want/it"
>
> This is fine for small deployments, but it’s a little annoying that it
> transfer *all* the files, even those that haven’t changed.  So I thought
> I could use rsync here, but it’s inconvenient that “guix pack” will do
> what it was designed for and produce a single file bundle.
>
> What do you think about adding an output format that is no format at all
> but a file enumeration printed to stdout?  That way I could use “guix
> pack” to produce a list of files to transfer and use that to transfer
> only the unchanged files.  Alternatively, perhaps we could have a
> “directory” format that merely copies (or links) the files to a new
> directory root.

You could untar the pack and rsync it.  We could have a ‘directory’
format but I’m afraid it would duplicate too much of the tarball code
(we’d have to check).

When you think about it, you could just as well have Guix on the other
side and then use ‘guix copy’… or maybe use Guix directly there.  (As in
‘guix pack --localstatedir -RR guix’.)

I know it’s not what you had in mind, but it seems to me that there’s a
continuum here, and maybe there’s just a gap to bridge to allow for
smoother workflows?

Thanks,
Ludo’.

Re: guix pack file enumerator?

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> “guix pack” is great for deployment of applications to servers that
>> don’t have Guix.  For a project I have a “deploy” target in my Makefile
>> that essentially does this:
>>
>>     cat $(shell guix pack -RR -e '(load "guix.scm")' -S /bin=bin) | ssh remote-server "tar xvzf - -C /where/i/want/it"
>>
>> This is fine for small deployments, but it’s a little annoying that it
>> transfer *all* the files, even those that haven’t changed.  So I thought
>> I could use rsync here, but it’s inconvenient that “guix pack” will do
>> what it was designed for and produce a single file bundle.
>>
>> What do you think about adding an output format that is no format at all
>> but a file enumeration printed to stdout?  That way I could use “guix
>> pack” to produce a list of files to transfer and use that to transfer
>> only the unchanged files.  Alternatively, perhaps we could have a
>> “directory” format that merely copies (or links) the files to a new
>> directory root.
>
> You could untar the pack and rsync it.  We could have a ‘directory’
> format but I’m afraid it would duplicate too much of the tarball code
> (we’d have to check).

I’m unpacking to rsync now, but it’s not great to duplicate files on the
local disk only to send them over the network.  The ideal workflow for
me would treat the store on my local machine as the source and the
remote machine as the target, without any copying from source to source
in between.

The local-unpack method also requires an empty directory to be
maintained; it needs to be emptied before the next time we try to unpack
anything.

Another downside is the creation of the tarball itself.  I don’t need it
but it’s still created only to be destroyed in the next step.  This is
unfortunate, because Guix could use a different mechanism to “check out”
files from the store to a new location.

> When you think about it, you could just as well have Guix on the other
> side and then use ‘guix copy’… or maybe use Guix directly there.  (As in
> ‘guix pack --localstatedir -RR guix’.)

I didn’t consider this, but yes: this is an option.  It requires the
target to be actively involved, though, which means that I need to spend
more time preparing it.  There’s a certain appeal in treating the target
as a mere dumping ground and coordinating everything from my development
machine.

> I know it’s not what you had in mind, but it seems to me that there’s a
> continuum here, and maybe there’s just a gap to bridge to allow for
> smoother workflows?

Probably.  The only difficulty I see is that none of the formats that
“guix pack” supports allow for arbitrary variations in the packing
process, such as using a different tool for the packing.  To be fair:
none of the currently supported formats need this level of flexibility.
But extending “guix pack” to domains that it was not primarily designed
to cover (such as deploying a pack before it is produced as a local
artifact) may require some of that flexibility.

-- 
Ricardo

Re: guix pack file enumerator?

[Ludovic Courtès]

Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

> I’m unpacking to rsync now, but it’s not great to duplicate files on the
> local disk only to send them over the network.  The ideal workflow for
> me would treat the store on my local machine as the source and the
> remote machine as the target, without any copying from source to source
> in between.
>
> The local-unpack method also requires an empty directory to be
> maintained; it needs to be emptied before the next time we try to unpack
> anything.
>
> Another downside is the creation of the tarball itself.  I don’t need it
> but it’s still created only to be destroyed in the next step.  This is
> unfortunate, because Guix could use a different mechanism to “check out”
> files from the store to a new location.

Here’s another idea: allowing ‘guix copy’ to talk to a “raw” remote
store—i.e., just /gnu/store + /var/guix/db accessed over SSH.

Just like we have (guix store ssh), we could implement raw store
operations over SSH using ‘remote-eval’ and (guix store database).  Hmm
that amounts to implementing a subset of the daemon.

Ludo’.

Re: guix pack file enumerator?

[Ryan Prior]

[-- Attachment #1: Type: text/plain, Size: 1251 bytes --]

On December 14, 2020, "Ludovic Courtès" <ludo@gnu.org> wrote:
> Here’s another idea: allowing ‘guix copy’ to talk to a “raw” remote
> store—i.e., just /gnu/store + /var/guix/db accessed over SSH.


> Hmm that amounts to implementing a subset of the daemon.

This reminds me of the much-loved "agentless" model of software
deployment as practiced by Ansible. [1] To provision, configure, and
deploy using Ansible all you need on the target machine is Python and
sshd. It copies its tools over, does its work, then cleans up after
itself.

If Guix could deploy to any machine with Guile + sshd that would be a
pretty compelling story. If you could also substitute perl or python or
something more ubiquitous, that would be even stronger, at the cost of
decreased code re-use. Best of all would be a mini-daemon we could
cross-compile for the target architecture, scp over as a single file,
execute there to carry out the rest of the task, then delete if no
longer needed. This approach is used to great effect by the Mgmt
automation tool [2].


[1] White paper (pdf, 5 pages)
https://www.ansible.com/hubfs/pdfs/Benefits-of-Agentless-WhitePaper.pdf
[2] https://github.com/purpleidea/mgmt/

[-- Attachment #2: Type: text/html, Size: 3593 bytes --]

Re: guix pack file enumerator?

[Nicolò Balzarotti]

> This reminds me of the much-loved "agentless" model of software
> deployment as practiced by Ansible.
> If Guix could deploy to any machine with Guile + sshd that would be a
> pretty compelling story.

Nice! This reminds me of nixos-infect [1].

But I think we don't even need guile, as we can guix pack -RR guix.  We
just need ssh and a "recent" linux kernel on the target machine.


[1] https://github.com/elitak/nixos-infect