Hi Guix, “guix pack” is great for deployment of applications to servers that don’t have Guix. For a project I have a “deploy” target in my Makefile that essentially does this: cat $(shell guix pack -RR -e '(load "guix.scm")' -S /bin=bin) | ssh remote-server "tar xvzf - -C /where/i/want/it" This is fine for small deployments, but it’s a little annoying that it transfer *all* the files, even those that haven’t changed. So I thought I could use rsync here, but it’s inconvenient that “guix pack” will do what it was designed for and produce a single file bundle. What do you think about adding an output format that is no format at all but a file enumeration printed to stdout? That way I could use “guix pack” to produce a list of files to transfer and use that to transfer only the unchanged files. Alternatively, perhaps we could have a “directory” format that merely copies (or links) the files to a new directory root. -- Ricardo
[-- Attachment #1: Type: text/plain, Size: 981 bytes --] On December 6, 2020, Ricardo Wurmus <rekado@elephly.net> wrote: > What do you think about adding an output format that is no format at > all > but a file enumeration printed to stdout? That way I could use “guix > pack” to produce a list of files to transfer and use that to transfer > only the unchanged files. Alternatively, perhaps we could have a > “directory” format that merely copies (or links) the files to a new > directory root. These suggestions comport nicely with the Unix-philosophy and something like that should probably be implemented for people who want to build their own deploy systems like you're describing. Additionally, I think that the use-case of "deploy this software to some server that doesn't have Guix installed" is important enough that we might want to support it explicitly such that "guix deploy" does the right thing, doing minimal work to bring the target deployment up-to- date. Is that feasible? [-- Attachment #2: Type: text/html, Size: 3293 bytes --]
Hi Ricardo, On Sun, 06 Dec 2020 at 10:53, Ricardo Wurmus <rekado@elephly.net> wrote: > What do you think about adding an output format that is no format at all > but a file enumeration printed to stdout? That way I could use “guix > pack” to produce a list of files to transfer and use that to transfer > only the unchanged files. Alternatively, perhaps we could have a > “directory” format that merely copies (or links) the files to a new > directory root. Well, I am not sure if it fits your use case but I am still annoyed to find the “profile/manifest“ path when I use: guix pack -f docker --save-provenance -m manifest.scm then extracts this “profile/manifest” needs: --8<---------------cut here---------------start------------->8--- docker export -o foo.tar <ID> tar -xf foo.tar $(tar -tf foo.tar | grep ’profile/manifest’) --8<---------------cut here---------------end--------------->8--- in order to rebuild an approximation of ’manifest.scm+channels.scm’ if it has been lost, for example to be able to re-run at anytime: --8<---------------cut here---------------start------------->8--- docker load < $(guix time-machine -C channels.scm \ -- pack -f docker \ -m manifest.scm) docker run -ti <TAG> --8<---------------cut here---------------end--------------->8--- For reference, see <https://yhetil.org/guix/86sgb9nt4k.fsf@gmail.com>. Therefore, I do not know if this output format is helping for this. Thanks, simon
Hi,
Ricardo Wurmus <rekado@elephly.net> skribis:
> “guix pack” is great for deployment of applications to servers that
> don’t have Guix. For a project I have a “deploy” target in my Makefile
> that essentially does this:
>
> cat $(shell guix pack -RR -e '(load "guix.scm")' -S /bin=bin) | ssh remote-server "tar xvzf - -C /where/i/want/it"
>
> This is fine for small deployments, but it’s a little annoying that it
> transfer *all* the files, even those that haven’t changed. So I thought
> I could use rsync here, but it’s inconvenient that “guix pack” will do
> what it was designed for and produce a single file bundle.
>
> What do you think about adding an output format that is no format at all
> but a file enumeration printed to stdout? That way I could use “guix
> pack” to produce a list of files to transfer and use that to transfer
> only the unchanged files. Alternatively, perhaps we could have a
> “directory” format that merely copies (or links) the files to a new
> directory root.
You could untar the pack and rsync it. We could have a ‘directory’
format but I’m afraid it would duplicate too much of the tarball code
(we’d have to check).
When you think about it, you could just as well have Guix on the other
side and then use ‘guix copy’… or maybe use Guix directly there. (As in
‘guix pack --localstatedir -RR guix’.)
I know it’s not what you had in mind, but it seems to me that there’s a
continuum here, and maybe there’s just a gap to bridge to allow for
smoother workflows?
Thanks,
Ludo’.
Ludovic Courtès <ludo@gnu.org> writes: > Hi, > > Ricardo Wurmus <rekado@elephly.net> skribis: > >> “guix pack” is great for deployment of applications to servers that >> don’t have Guix. For a project I have a “deploy” target in my Makefile >> that essentially does this: >> >> cat $(shell guix pack -RR -e '(load "guix.scm")' -S /bin=bin) | ssh remote-server "tar xvzf - -C /where/i/want/it" >> >> This is fine for small deployments, but it’s a little annoying that it >> transfer *all* the files, even those that haven’t changed. So I thought >> I could use rsync here, but it’s inconvenient that “guix pack” will do >> what it was designed for and produce a single file bundle. >> >> What do you think about adding an output format that is no format at all >> but a file enumeration printed to stdout? That way I could use “guix >> pack” to produce a list of files to transfer and use that to transfer >> only the unchanged files. Alternatively, perhaps we could have a >> “directory” format that merely copies (or links) the files to a new >> directory root. > > You could untar the pack and rsync it. We could have a ‘directory’ > format but I’m afraid it would duplicate too much of the tarball code > (we’d have to check). I’m unpacking to rsync now, but it’s not great to duplicate files on the local disk only to send them over the network. The ideal workflow for me would treat the store on my local machine as the source and the remote machine as the target, without any copying from source to source in between. The local-unpack method also requires an empty directory to be maintained; it needs to be emptied before the next time we try to unpack anything. Another downside is the creation of the tarball itself. I don’t need it but it’s still created only to be destroyed in the next step. This is unfortunate, because Guix could use a different mechanism to “check out” files from the store to a new location. > When you think about it, you could just as well have Guix on the other > side and then use ‘guix copy’… or maybe use Guix directly there. (As in > ‘guix pack --localstatedir -RR guix’.) I didn’t consider this, but yes: this is an option. It requires the target to be actively involved, though, which means that I need to spend more time preparing it. There’s a certain appeal in treating the target as a mere dumping ground and coordinating everything from my development machine. > I know it’s not what you had in mind, but it seems to me that there’s a > continuum here, and maybe there’s just a gap to bridge to allow for > smoother workflows? Probably. The only difficulty I see is that none of the formats that “guix pack” supports allow for arbitrary variations in the packing process, such as using a different tool for the packing. To be fair: none of the currently supported formats need this level of flexibility. But extending “guix pack” to domains that it was not primarily designed to cover (such as deploying a pack before it is produced as a local artifact) may require some of that flexibility. -- Ricardo
Hi,
Ricardo Wurmus <rekado@elephly.net> skribis:
> I’m unpacking to rsync now, but it’s not great to duplicate files on the
> local disk only to send them over the network. The ideal workflow for
> me would treat the store on my local machine as the source and the
> remote machine as the target, without any copying from source to source
> in between.
>
> The local-unpack method also requires an empty directory to be
> maintained; it needs to be emptied before the next time we try to unpack
> anything.
>
> Another downside is the creation of the tarball itself. I don’t need it
> but it’s still created only to be destroyed in the next step. This is
> unfortunate, because Guix could use a different mechanism to “check out”
> files from the store to a new location.
Here’s another idea: allowing ‘guix copy’ to talk to a “raw” remote
store—i.e., just /gnu/store + /var/guix/db accessed over SSH.
Just like we have (guix store ssh), we could implement raw store
operations over SSH using ‘remote-eval’ and (guix store database). Hmm
that amounts to implementing a subset of the daemon.
Ludo’.
[-- Attachment #1: Type: text/plain, Size: 1251 bytes --] On December 14, 2020, "Ludovic Courtès" <ludo@gnu.org> wrote: > Here’s another idea: allowing ‘guix copy’ to talk to a “raw” remote > store—i.e., just /gnu/store + /var/guix/db accessed over SSH. > Hmm that amounts to implementing a subset of the daemon. This reminds me of the much-loved "agentless" model of software deployment as practiced by Ansible. [1] To provision, configure, and deploy using Ansible all you need on the target machine is Python and sshd. It copies its tools over, does its work, then cleans up after itself. If Guix could deploy to any machine with Guile + sshd that would be a pretty compelling story. If you could also substitute perl or python or something more ubiquitous, that would be even stronger, at the cost of decreased code re-use. Best of all would be a mini-daemon we could cross-compile for the target architecture, scp over as a single file, execute there to carry out the rest of the task, then delete if no longer needed. This approach is used to great effect by the Mgmt automation tool [2]. [1] White paper (pdf, 5 pages) https://www.ansible.com/hubfs/pdfs/Benefits-of-Agentless-WhitePaper.pdf [2] https://github.com/purpleidea/mgmt/ [-- Attachment #2: Type: text/html, Size: 3593 bytes --]
> This reminds me of the much-loved "agentless" model of software > deployment as practiced by Ansible. > If Guix could deploy to any machine with Guile + sshd that would be a > pretty compelling story. Nice! This reminds me of nixos-infect [1]. But I think we don't even need guile, as we can guix pack -RR guix. We just need ssh and a "recent" linux kernel on the target machine. [1] https://github.com/elitak/nixos-infect