Guix "ops"

Guix "ops"

[David Thompson]

Hey folks,

I'm writing this in an attempt to "think out loud" about a deployment
automation tool for GuixSD.

Guix needs a NixOps[0] equivalent.  NixOps is the Nix project's
deployment automation tool.  I read the source code[1] a bit, and here's
what I've learned about it:

* The central data type is a "deployment", which is a Nix expression
  consisting of one or more named OS configs
  
* The OS configs contain extra data that specifies the target platform:
  VirtualBox, EC2, NixOS container, etc.  Each platform implements the
  generic MachineDefinition and MachineState interfaces.

* The 'nixops' command is stateful.  It stores necessary state about
  deployed systems in a special directory ($HOME/.nixops by default),
  such as the IP addresses of the deployed systems.  Because of this,
  each deployment needs a unique identifier.

* Deployments may be parameterized by values known only at deploy time.
  This covers cases such as a web application server needing to know the
  IP address of the database server.

* There are useful subcommands to check the status of each system or ssh
  into one of them.

Here's a rough outline of how I'm thinking of implementing similar
features in Guix:

* Introduce new data types:

  * <platform>: The generic interface type for implementing deployment
    targets.  Its fields hold procedures for various actions like
    'provision', 'destroy', 'boot', or 'reboot'.  I haven't determined
    the precise list of actions needed, but it will become apparent as
    platforms are added.
    
  * <machine>: Describes a single system in the deployment.  Contains a
    name string, an <operating-system>, and a <platform>.

  * <deployment>: Contains a name string and a list of <machine> and
    procedures.  Procedures in the list should accept an argument
    containing the deployment results of the non-parameterized machines
    and return a <machine>.

* Use a simple s-exp deployment state format.  Store the state in
  $HOME/.config/guix by default.

* Implement a 'guix ops' subcommand roughly the same actions as NixOps:
  create, deploy, start, stop, destroy, list, info, check, ssh, etc.

* The bulk of the work will be creating <platform> objects that actually
  provision various types of resources.  For prototyping, a
  'local-vm-platform' would be enough to test that the whole system
  works.

Anyone want to join in on this brainstorming party?  Your thoughts are
appreciated!

[0] http://nixos.org/nixops/
[1] https://github.com/NixOS/nixops

-- 
David Thompson
Web Developer - Free Software Foundation - http://fsf.org
GPG Key: 0FF1D807
Support the FSF: https://fsf.org/donate

Re: Guix "ops"

[Ludovic Courtès]

David Thompson <dthompson2@worcester.edu> skribis:

> I'm writing this in an attempt to "think out loud" about a deployment
> automation tool for GuixSD.

Yay, good idea!

> * The central data type is a "deployment", which is a Nix expression
>   consisting of one or more named OS configs
>   
> * The OS configs contain extra data that specifies the target platform:
>   VirtualBox, EC2, NixOS container, etc.  Each platform implements the
>   generic MachineDefinition and MachineState interfaces.
>
> * The 'nixops' command is stateful.  It stores necessary state about
>   deployed systems in a special directory ($HOME/.nixops by default),
>   such as the IP addresses of the deployed systems.  Because of this,
>   each deployment needs a unique identifier.

Oh, I remember “charon create” creating this ‘network.json’ file
containing the list of machines and the file names of the Nix expression
used to create that deployment.

I’m not 100% clear on why this state needs to be stored; it seems more
like a cache to me, no?  That is, Charon/NixOps can always recreate it
from the source Nix expressions.

> * Deployments may be parameterized by values known only at deploy time.
>   This covers cases such as a web application server needing to know the
>   IP address of the database server.
>
> * There are useful subcommands to check the status of each system or ssh
>   into one of them.
>
> Here's a rough outline of how I'm thinking of implementing similar
> features in Guix:
>
> * Introduce new data types:
>
>   * <platform>: The generic interface type for implementing deployment
>     targets.  Its fields hold procedures for various actions like
>     'provision', 'destroy', 'boot', or 'reboot'.  I haven't determined
>     the precise list of actions needed, but it will become apparent as
>     platforms are added.

OK.

>   * <machine>: Describes a single system in the deployment.  Contains a
>     name string, an <operating-system>, and a <platform>.

Yes (it’s best to keep it separate from <operating-system>; in NixOps
it’s a ‘deployment’ attribute in the OS attribute set.)

>   * <deployment>: Contains a name string and a list of <machine> and
>     procedures.  Procedures in the list should accept an argument
>     containing the deployment results of the non-parameterized machines
>     and return a <machine>.

OK.

> * Use a simple s-exp deployment state format.  Store the state in
>   $HOME/.config/guix by default.

Or ~/.cache/guix?

> * Implement a 'guix ops' subcommand roughly the same actions as NixOps:
>   create, deploy, start, stop, destroy, list, info, check, ssh, etc.
>
> * The bulk of the work will be creating <platform> objects that actually
>   provision various types of resources.  For prototyping, a
>   'local-vm-platform' would be enough to test that the whole system
>   works.

Sure.

> Anyone want to join in on this brainstorming party?  Your thoughts are
> appreciated!

It seems you already have all the requirements and design options
in mind!

Thanks,
Ludo’.

Re: Guix "ops"

[David Thompson]

Ludovic Courtès <ludo@gnu.org> writes:

> David Thompson <dthompson2@worcester.edu> skribis:
>
>> * The 'nixops' command is stateful.  It stores necessary state about
>>   deployed systems in a special directory ($HOME/.nixops by default),
>>   such as the IP addresses of the deployed systems.  Because of this,
>>   each deployment needs a unique identifier.
>
> Oh, I remember “charon create” creating this ‘network.json’ file
> containing the list of machines and the file names of the Nix expression
> used to create that deployment.
>
> I’m not 100% clear on why this state needs to be stored; it seems more
> like a cache to me, no?  That is, Charon/NixOps can always recreate it
> from the source Nix expressions.

The state that I am concerned with are the details of the resources that
have been provisioned by 'guix ops'.  For example, in a system that
provisions VMs on behalf of the user, the IP address of the machine
isn't known until the provisioning has happened.  This detail needs to
be saved so that 'guix ops' knows how to perform future operations on
the machine.

Caching the file names of the expressions with a unique key seems hacky
to me, and I don't want to copy that without good reason.

>>   * <machine>: Describes a single system in the deployment.  Contains a
>>     name string, an <operating-system>, and a <platform>.
>
> Yes (it’s best to keep it separate from <operating-system>; in NixOps
> it’s a ‘deployment’ attribute in the OS attribute set.)

Yes, though I do have a question here.  Some platforms don't do anything
with a bootloader, or in the case of containers (looking towards the
future here) the 'kernel' field will be benign as the system shares the
kernel of its host.  I'm not sure how to deal with this extraneous
information.  In the case of the 'bootloader', it currently must be
specified, so the user would have to input bootloader details that are
irrelevant.  Thoughts?

>> * Use a simple s-exp deployment state format.  Store the state in
>>   $HOME/.config/guix by default.
>
> Or ~/.cache/guix?

Sure, that makes more sense.

>> Anyone want to join in on this brainstorming party?  Your thoughts are
>> appreciated!
>
> It seems you already have all the requirements and design options
> in mind!

Thanks, but things are still a bit fuzzy so any design other design
considerations are much appreciated. :)

So far, I have created the barebones 'guix ops' subcommand and defined
the new data types.  One can run 'guix ops build deployment.scm' to
build all of the machines in a deployment, but no other subcommands have
been implemented.  For the prototype, I envision 'guix ops deploy
local-vm-deployment.scm' to work much like 'guix system vm system.scm',
except that it builds and boots multiple VMs.

-- 
David Thompson
Web Developer - Free Software Foundation - http://fsf.org
GPG Key: 0FF1D807
Support the FSF: https://fsf.org/donate

Re: Guix "ops"

[Ludovic Courtès]

David Thompson <dthompson2@worcester.edu> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> David Thompson <dthompson2@worcester.edu> skribis:
>>
>>> * The 'nixops' command is stateful.  It stores necessary state about
>>>   deployed systems in a special directory ($HOME/.nixops by default),
>>>   such as the IP addresses of the deployed systems.  Because of this,
>>>   each deployment needs a unique identifier.
>>
>> Oh, I remember “charon create” creating this ‘network.json’ file
>> containing the list of machines and the file names of the Nix expression
>> used to create that deployment.
>>
>> I’m not 100% clear on why this state needs to be stored; it seems more
>> like a cache to me, no?  That is, Charon/NixOps can always recreate it
>> from the source Nix expressions.
>
> The state that I am concerned with are the details of the resources that
> have been provisioned by 'guix ops'.  For example, in a system that
> provisions VMs on behalf of the user, the IP address of the machine
> isn't known until the provisioning has happened.  This detail needs to
> be saved so that 'guix ops' knows how to perform future operations on
> the machine.

Hmm, I see.

> Caching the file names of the expressions with a unique key seems hacky
> to me, and I don't want to copy that without good reason.

I agree.  There must be a way to at least to a one-way mapping from the
<machine> object to the corresponding key in the cache.

>>>   * <machine>: Describes a single system in the deployment.  Contains a
>>>     name string, an <operating-system>, and a <platform>.
>>
>> Yes (it’s best to keep it separate from <operating-system>; in NixOps
>> it’s a ‘deployment’ attribute in the OS attribute set.)
>
> Yes, though I do have a question here.  Some platforms don't do anything
> with a bootloader, or in the case of containers (looking towards the
> future here) the 'kernel' field will be benign as the system shares the
> kernel of its host.  I'm not sure how to deal with this extraneous
> information.  In the case of the 'bootloader', it currently must be
> specified, so the user would have to input bootloader details that are
> irrelevant.  Thoughts?

What about simply ignoring them?  ‘guix system vm’ ignores the
bootloader, for instance, unless it is passed --full-boot.

> So far, I have created the barebones 'guix ops' subcommand and defined
> the new data types.  One can run 'guix ops build deployment.scm' to
> build all of the machines in a deployment, but no other subcommands have
> been implemented.  For the prototype, I envision 'guix ops deploy
> local-vm-deployment.scm' to work much like 'guix system vm system.scm',
> except that it builds and boots multiple VMs.

Sounds good to me.

The thing that needs more thought is how to handle things like
networking.  I guess the tool would need to automatically add, say, a
‘dhcp-client-service’ to each <operating-system> object, and to remove
existing services that provide ‘networking’, and probably fiddle with
other parameters such as the initrd and file systems.  (Similar to what
‘virtualized-operating-system’ does in (gnu system vm).)

In effect, one would not be deploying the exact OS that is specified,
but rather a variant automatically customized for the deployment target.

So probably each <deployment> object must specified an
<operating-system> -> <operating-system> procedure that does this
transformation.

WDYT?

BTW, I’d prefer something like ‘guix deploy’ over ‘guix ops’, but then
another name would need to be found for the ‘deploy’ sub-command, maybe
‘realize’?

Thanks,
Ludo’.

Re: Guix "ops"

[Carlos Sosa]

Hi David and Ludovic,

I've been doing configuration management in GuixSD with propellor by
Joey Hess. https://propellor.branchable.com/

While it works great, I would prefer any tool that works with Guile
syntax.

ludo@gnu.org (Ludovic Courtès) writes:

> BTW, I’d prefer something like ‘guix deploy’ over ‘guix ops’, but then
> another name would need to be found for the ‘deploy’ sub-command, maybe
> ‘realize’?

  I like the idea of 'guix deploy', and maybe something to propagates
  given configuration files, like 'guix config prepare' and later 'guix
  config apply'.

Now, how can I contribute? work the guix command?

Let me know if you have a specific repository or place to find any work
done on this.

- Carlos

Re: Guix "ops"

[David Thompson]

Carlos Sosa <gnusosa@gnusosa.net> writes:

> Hi David and Ludovic,
>
> I've been doing configuration management in GuixSD with propellor by
> Joey Hess. https://propellor.branchable.com/

That's really cool.  Would you please share your configuration files?
I'd love to see what that looks like.

> While it works great, I would prefer any tool that works with Guile
> syntax.

Yes, we should have our own tool for this purpose that has excellent
integration with the rest of the system.

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> BTW, I’d prefer something like ‘guix deploy’ over ‘guix ops’, but then
>> another name would need to be found for the ‘deploy’ sub-command, maybe
>> ‘realize’?
>
>   I like the idea of 'guix deploy', and maybe something to propagates
>   given configuration files, like 'guix config prepare' and later 'guix
>   config apply'.

The only configuration we'd apply is a full system configuration, a la
'guix system reconfigure'.

> Now, how can I contribute? work the guix command?
>
> Let me know if you have a specific repository or place to find any work
> done on this.

I should create a "wip-deploy" branch in our git repository that you
could submit patches for.  I need to do a bit more work to get the very
basics working so we have a foundation to build on, though.

The other emails in this thread detail my basic strategy.  Here's an
example of a 'guix deploy' config that actually works right now:
http://paste.lisp.org/display/147741

I'm happy to discuss this more and figure out how to get you involved!
Thanks for expressing your interest.

-- 
David Thompson
Web Developer - Free Software Foundation - http://fsf.org
GPG Key: 0FF1D807
Support the FSF: https://fsf.org/donate

Re: Guix "ops"

[Ludovic Courtès]

David Thompson <dthompson2@worcester.edu> skribis:

> I should create a "wip-deploy" branch in our git repository that you
> could submit patches for.  I need to do a bit more work to get the very
> basics working so we have a foundation to build on, though.

That would be nice, feel free to do push a branch whenever you deem it
appropriate!

Ludo’.

Re: Guix "ops"

[Christopher Allan Webber]

Ludovic Courtès writes:

> David Thompson <dthompson2@worcester.edu> skribis:
>
>> I should create a "wip-deploy" branch in our git repository that you
>> could submit patches for.  I need to do a bit more work to get the very
>> basics working so we have a foundation to build on, though.
>
> That would be nice, feel free to do push a branch whenever you deem it
> appropriate!
>
> Ludo’.

Sooner rather than later I hope!  I am looking forward to it. :)

Re: Guix "ops"

[David Thompson]

Hello again Carlos,

Carlos Sosa <gnusosa@gnusosa.net> writes:

>   I like the idea of 'guix deploy', and maybe something to propagates
>   given configuration files, like 'guix config prepare' and later 'guix
>   config apply'.
>
> Now, how can I contribute? work the guix command?
>
> Let me know if you have a specific repository or place to find any work
> done on this.

I have just pushed a new branch called "wip-deploy" to the official guix
repository.  Since this branch is prefixed with "wip-", expect it to be
rebased frequently.  There's not much code here yet, but a very simple
prototype has been implemented that supports the creation of local QEMU
VMs.

To take it for a spin, add something like this to a file, let's call it
"deployment.scm":

    (use-modules (gnu) (guix gexp))
    (use-service-modules databases)
    (use-package-modules web databases)
    
    (define dummy-fs
      (file-system
        (mount-point "/")
        (device "dummy")
        (type "dummy")))
    
    (define grub
      (grub-configuration (device "/dev/sda")))
    
    (define timezone "America/New_York")
    (define locale "en_US.UTF-8")
    
    ;; TODO: Needs nginx-service.
    (define web-server
      (machine
       (name "web-server")
       (system (operating-system
                 (host-name "web-server")
                 (timezone timezone)
                 (locale locale)
                 (bootloader grub)
                 (file-systems
                  (list dummy-fs %binary-format-file-system))
                 (packages
                  (cons nginx %base-packages))))
       (platform (local-vm #:ip-address "10.0.2.10"))))
    
    (define db-server
      (machine
       (name "db-server")
       (system (operating-system
                 (host-name "db-server")
                 (timezone timezone)
                 (locale locale)
                 (bootloader grub)
                 (file-systems
                  (list dummy-fs %binary-format-file-system))
                 (services
                  (cons (postgresql-service)
                        %base-services))
                 (packages (cons postgresql %base-packages))))
       (platform (local-vm #:ip-address "10.0.2.11"))))
    
    (deployment
     (name "test-deployment")
     (machines (list web-server db-server)))
    
Then run the following from your git checkout:

    ./pre-inst-env guix deploy spawn /path/to/deployment.scm

One caveat: Make sure the file name uses an absolute path for now.  I
haven't cleaned up the code enough to deal with relative file names.

If the command is successful, you will see 2 QEMU windows open up, one
for the web server and one for the database server.  Pretty neat, eh? :)

Now, there's still much work to be done.  Spawning local, temporary VMs
has gotten me over some initial hurdles, but now we need to write a
platform adapter for something more serious so that we can determine the
requirements for "real world" deployment scenarios.  Perhaps we should
look into writing an OpenStack adapter.

There's also unanswered questions like: How should we keep track of
state?  How do we reconfigure already deployed machines?  How do we shut
down a deployment and unprovision the resources it used?  Basically, how
many hooks does the <platform> record type need to cover everything?

Thoughts and help very much wanted!

-- 
David Thompson
GPG Key: 0FF1D807

Re: Guix "ops"

[Ludovic Courtès]

David Thompson <dthompson2@worcester.edu> skribis:

> To take it for a spin, add something like this to a file, let's call it
> "deployment.scm":
>
>     (use-modules (gnu) (guix gexp))
>     (use-service-modules databases)
>     (use-package-modules web databases)
>     
>     (define dummy-fs
>       (file-system
>         (mount-point "/")
>         (device "dummy")
>         (type "dummy")))
>     
>     (define grub
>       (grub-configuration (device "/dev/sda")))
>     
>     (define timezone "America/New_York")
>     (define locale "en_US.UTF-8")
>     
>     ;; TODO: Needs nginx-service.
>     (define web-server
>       (machine
>        (name "web-server")
>        (system (operating-system
>                  (host-name "web-server")
>                  (timezone timezone)
>                  (locale locale)
>                  (bootloader grub)
>                  (file-systems
>                   (list dummy-fs %binary-format-file-system))
>                  (packages
>                   (cons nginx %base-packages))))
>        (platform (local-vm #:ip-address "10.0.2.10"))))
>     
>     (define db-server
>       (machine
>        (name "db-server")
>        (system (operating-system
>                  (host-name "db-server")
>                  (timezone timezone)
>                  (locale locale)
>                  (bootloader grub)
>                  (file-systems
>                   (list dummy-fs %binary-format-file-system))
>                  (services
>                   (cons (postgresql-service)
>                         %base-services))
>                  (packages (cons postgresql %base-packages))))
>        (platform (local-vm #:ip-address "10.0.2.11"))))
>     
>     (deployment
>      (name "test-deployment")
>      (machines (list web-server db-server)))
>     
> Then run the following from your git checkout:
>
>     ./pre-inst-env guix deploy spawn /path/to/deployment.scm

Woow, that’s very nice.  The declarations are straightforward and
immediately understandable, I think, which is cool.

Perhaps one addition eventually would be to allow IPs to be
automatically allocated and have host name lookup DTRT in each VM.

> Now, there's still much work to be done.  Spawning local, temporary VMs
> has gotten me over some initial hurdles, but now we need to write a
> platform adapter for something more serious so that we can determine the
> requirements for "real world" deployment scenarios.  Perhaps we should
> look into writing an OpenStack adapter.

As discussed on IRC, I was unsure about OpenStack, but I’ll trust your
judgment.  Maybe Cyril can comment?

Thanks,
Ludo’.

Re: Guix "ops"

[David Thompson]

Ludovic Courtès <ludo@gnu.org> writes:

> Perhaps one addition eventually would be to allow IPs to be
> automatically allocated and have host name lookup DTRT in each VM.

Do you have any idea how we could do that for local VMs?  There's no
daemon managing the provision of these resources, so I don't know what
strategy can be used to automatically allocate static IPs.

The automagic hostname lookup part is particularly interesting to me.  A
more complete deployment configuration would have the web server
dependent on the db server.  I originally intended to handle this by
delaying the creation of the web server until after the db server was
made, and invoking a procedure that accepted the db server's state as
input and output the correct configuration for the web server.

Thinking out loud here: Maybe 'guix deploy' can kick off the
provisioning for all machines first, and afterwards the OS configs can
be altered to include the correct /etc/hosts file.

> As discussed on IRC, I was unsure about OpenStack, but I’ll trust your
> judgment.  Maybe Cyril can comment?

I threw out OpenStack because it's a self-hostable, free software VM
platform.  I'm open to any other platforms that will exercise the full
range of capabilities that 'guix deploy' needs to be useful.

-- 
David Thompson
GPG Key: 0FF1D807

Re: Guix "ops"

[Carlos Sosa]

David Thompson <dthompson2@worcester.edu> writes:

> Thinking out loud here: Maybe 'guix deploy' can kick off the
> provisioning for all machines first, and afterwards the OS configs can
> be altered to include the correct /etc/hosts file.

  I like the idea of `guix deploy` with a minor change where we add
  `guix deploy machine`, but can that be a command that calls separate
  steps like `guix provision machine` and then `guix set-config machine`
  or something similar. My intention with that, is that if the command
  `guix set-config machine` fails at least `guix deploy machine` worked
  and you can SSH to that machine or debug why did the OS configs
  failed. To deploy all of our machines in #machines-list we can do
  `guix deploy` where it defaults to `guix deploy all`. I guess this
  hints more on the usability perspective, but I think provisioning the
  OS and the configuration provision should be separate tasks. I will
  try to develop that soon. I know a patch says more than words. :)

> I threw out OpenStack because it's a self-hostable, free software VM
> platform.  I'm open to any other platforms that will exercise the full
> range of capabilities that 'guix deploy' needs to be useful.

  I've been using LXC - https://en.wikipedia.org/wiki/LXC to work around
  with Guix and play with substitutes and offloading. It's been pretty
  straight forward and clean, I don't know if that would be something to
  consider.

  - Carlos

Re: Guix "ops"

[Thompson, David]

On Wed, May 27, 2015 at 2:47 PM, Carlos Sosa <gnusosa@gnusosa.net> wrote:
> David Thompson <dthompson2@worcester.edu> writes:
>
>> Thinking out loud here: Maybe 'guix deploy' can kick off the
>> provisioning for all machines first, and afterwards the OS configs can
>> be altered to include the correct /etc/hosts file.
>
>   I like the idea of `guix deploy` with a minor change where we add
>   `guix deploy machine`, but can that be a command that calls separate
>   steps like `guix provision machine` and then `guix set-config machine`
>   or something similar. My intention with that, is that if the command
>   `guix set-config machine` fails at least `guix deploy machine` worked
>   and you can SSH to that machine or debug why did the OS configs
>   failed. To deploy all of our machines in #machines-list we can do
>   `guix deploy` where it defaults to `guix deploy all`. I guess this
>   hints more on the usability perspective, but I think provisioning the
>   OS and the configuration provision should be separate tasks. I will
>   try to develop that soon. I know a patch says more than words. :)

I intend to have several subcommands for 'guix deploy' that perform
various subsets of the deployment "phases".  I don't think your
use-case for "set-config" phase makes sense, because the entire system
is configured in a single operation.  The system would have to boot a
valid GuixSD configuration at least once in order for SSH access to be
available.  Now, a re-deploy (similar to 'guix system reconfigure')
may fail, and that could be rolled back in case of problems.

>> I threw out OpenStack because it's a self-hostable, free software VM
>> platform.  I'm open to any other platforms that will exercise the full
>> range of capabilities that 'guix deploy' needs to be useful.
>
>   I've been using LXC - https://en.wikipedia.org/wiki/LXC to work around
>   with Guix and play with substitutes and offloading. It's been pretty
>   straight forward and clean, I don't know if that would be something to
>   consider.

Cool.  I'd be interested in reading the steps you took to deploy a
GuixSD system with LXC.  FYI, I'm (slowly) working on a pure Guile
container implementation for inclusion in Guix in the future.

- Dave

Re: Guix "ops"

[Ludovic Courtès]

David Thompson <dthompson2@worcester.edu> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Perhaps one addition eventually would be to allow IPs to be
>> automatically allocated and have host name lookup DTRT in each VM.
>
> Do you have any idea how we could do that for local VMs?  There's no
> daemon managing the provision of these resources, so I don't know what
> strategy can be used to automatically allocate static IPs.

QEMU allows you to specify the guest’s IP, I think, and apparently it
can create VLANs and connect several unprivileged QEMU instances
together via -net socket (info "(qemu-doc) sec_invocation").

Things like libvirt probably provide a higher-level interface to that.
(I don’t know if it justifies the extra dependency.)

> The automagic hostname lookup part is particularly interesting to me.  A
> more complete deployment configuration would have the web server
> dependent on the db server.  I originally intended to handle this by
> delaying the creation of the web server until after the db server was
> made, and invoking a procedure that accepted the db server's state as
> input and output the correct configuration for the web server.

I’ve seen that Docker can do that.  ;-)  IIRC it populates /etc/hosts in
each container.  That’s something we could do.  Another possibility
would be to rely on mDNS.

> Thinking out loud here: Maybe 'guix deploy' can kick off the
> provisioning for all machines first, and afterwards the OS configs can
> be altered to include the correct /etc/hosts file.

The transform procedure could force the right /etc/hosts in each OS, I
suppose?

Thanks,
Ludo’.

Re: Guix "ops"

[Thompson, David]

On Wed, May 27, 2015 at 3:41 PM, Ludovic Courtès <ludo@gnu.org> wrote:
> David Thompson <dthompson2@worcester.edu> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>>
>>> Perhaps one addition eventually would be to allow IPs to be
>>> automatically allocated and have host name lookup DTRT in each VM.
>>
>> Do you have any idea how we could do that for local VMs?  There's no
>> daemon managing the provision of these resources, so I don't know what
>> strategy can be used to automatically allocate static IPs.
>
> QEMU allows you to specify the guest’s IP, I think, and apparently it
> can create VLANs and connect several unprivileged QEMU instances
> together via -net socket (info "(qemu-doc) sec_invocation").
>
> Things like libvirt probably provide a higher-level interface to that.
> (I don’t know if it justifies the extra dependency.)

If libvirt's API was really useful, perhaps it could be an optional
dependency for users that want to deploy QEMU VMs?

>> The automagic hostname lookup part is particularly interesting to me.  A
>> more complete deployment configuration would have the web server
>> dependent on the db server.  I originally intended to handle this by
>> delaying the creation of the web server until after the db server was
>> made, and invoking a procedure that accepted the db server's state as
>> input and output the correct configuration for the web server.
>
> I’ve seen that Docker can do that.  ;-)  IIRC it populates /etc/hosts in
> each container.  That’s something we could do.  Another possibility
> would be to rely on mDNS.

If Docker can do it, so can we! :)

>> Thinking out loud here: Maybe 'guix deploy' can kick off the
>> provisioning for all machines first, and afterwards the OS configs can
>> be altered to include the correct /etc/hosts file.
>
> The transform procedure could force the right /etc/hosts in each OS, I
> suppose?

Yes.  Perhaps the extensibility I had in mind could be better achieved
by allowing additional, user specified transformations in the machine
declaration.

Thanks for your input!

- Dave

OpenStack and GuixOps (was: Re: Guix "ops")

[Christopher Allan Webber]

David Thompson writes:

>> As discussed on IRC, I was unsure about OpenStack, but I’ll trust your
>> judgment.  Maybe Cyril can comment?
>
> I threw out OpenStack because it's a self-hostable, free software VM
> platform.  I'm open to any other platforms that will exercise the full
> range of capabilities that 'guix deploy' needs to be useful.

I'm starting to put some time into GuixOps and seeing how I can help
today.  Dave has pointed out that for now we should focus on testing
with VMs and containers, but that OpenStack will still be a useful
long-term goal.  As such, I talked to a friend of mine (Boris Bobrov)
who works on OpenStack as part of his dayjob.  He gave me this advice
and okayed me to posting the conversation to this list:

  <paroneayea> I'm interested in helping with the guixops adapter to             
               openstack, but I don't know really where to get started in        
               openstack land
  <paroneayea> do you have a good pointer on where to start?                     
  <breton> pong
  <breton> well, "openstack 101" will give some theoretical overview
  <breton> after that -- http://docs.openstack.org/developer/devstack/. Devstack   
           is a bash script that sets up bleeding edge openstack for you           
  <breton> big red warning: do not set it up on your laptop directly. Better       
           install a minimal ubuntu/centos in a vm and run devstack there          
  <breton> since you want to do something with guix, my guess is that you need     
           to pay attention to the following openstack components: nova (the       
           thing that brings up a VM), glance (image registry, stores metadata     
           about objects), swift (stores data), maybe murano (it my company's      
           component, it has something to do with app deployment to VMs)           
  <breton> official docs are very good too, btw: http://docs.openstack.org/        
  <breton> a very brief overview of what's going on in openstack: you tell nova    
           to bring up a vm, nova fetches image from glance (glance fetches it     
           from swift or from some other storage), fetches networking info from    
           neutron, fetches list of disks from cinder, connects it alltogether     
           and gives you a vm. Every operation if authenticated by keystone.       

It looks like Nova (OpenStack Compute) is the main thing we'll end up
targeting.  Some links:

 - Nix peoples' wiki page on the subject:
     https://nixos.org/wiki/NixOS_and_OpenStack_Compute
 - Nova's main page:
     http://www.openstack.org/software/openstack-compute/
 - Python library, which we may want to examine to make a Guile library
   equiv:
     https://github.com/openstack/python-novaclient

I figured it's best to document this on the list while I'm looking into
it.  Hope that helps!

 - Chris

Re: OpenStack and GuixOps (was: Re: Guix "ops")

[Ian Denhardt]

[-- Attachment #1: Type: text/plain, Size: 239 bytes --]

Quoting Christopher Allan Webber (2015-07-09 14:27:04)
>  ...

I'd throw heat[1] into the list of things to read up on, but otherwise 
this sounds like fairly good/complete advice.

-Ian

[1]: https://wiki.openstack.org/wiki/Heat

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAABCAAGBQJVnyuOAAoJEPZUuMfUyjy4q+cP/jDeCBXh+DxQL8LZLPbZlm7C
YFe+06wD7bdSKEBFUdhpggN9ItVGtzQ3ZXRDx87zOs6x4PjUgccWJXBxiCzJY1i4
FFIluUvbLrBoTA8fIIVI18wR6k3dYelNlNOfoEKQJtaD42N+RUWdbq1J3afXkyEM
5PZ5Tng6WMgYvW2nDoGtKZ7DZ3ne1RUWeCR+g/EPeyce8/5baOvu6lI7nR32haCC
R9Hm8EBWQWB3jr2cX+nVZsKla9bqi94WjwcqGkadDXl3qyZwQYiyRMHle1TJ6KR0
UQpVg/t98Lpfll4XXR+34DO2fMaWaTGqq0F0RZy4hZRSdzYKpjHDbJ+rUxNmDamm
l10bHQvvM486XvH+zWGgzj0WE+8JzIgNpofi3uAhd788hckgTuWnQBg3RJ4shjwn
9z5R8RVSRDI6sgni/V3RpjGaEtg/2nxoCmv2cNufFY/ft/ufjDyeUT6RLz1/hOHK
6zMbk93t/I0viZjXFXAGc72Q1c3ZdoNr/0TRVU1fsTGG+sIaqrNf5rfNLnjQ4yIu
TfHET/jGdThoRDO+vCu3P4k2hxVKDlXf0e/hMm8lrQeNWZJZTo4r+0KL7qDVn6q/
ShOR6qNP7bA7PIjZOFMwpP8qB1jymuCX8OUOG723H+fe9xHbO7kkgexnhBlFadK7
QwGOVLTlFIwDIk73s+gK
=nuaF
-----END PGP SIGNATURE-----

Re: OpenStack and GuixOps

[Ludovic Courtès]

Great, thanks for looking into this!

Ludo’.

Re: Guix "ops"

[Pjotr Prins]

> There's also unanswered questions like: How should we keep track of
> state?  How do we reconfigure already deployed machines?  How do we shut
> down a deployment and unprovision the resources it used?  Basically, how
> many hooks does the <platform> record type need to cover everything?

The current 'deploy' basically generates a machine from scratch -
there is some similar funtionality in Nix. I read the code and it is
pretty simplistic for now which is possible when you generate a
machine once.

What I would like to have is a state-machine that can be rerun within
an installed deploy in the vein of Puppet/Chef/cfengine but without
the complicated setup of a client/server model (the server 'model'
should simply be a git repository with branches).

That means that after 'deploy' we can run the state and it
checks/updates files that may have been changed. I use something like
that to run my hosts.allow configuration (for example). When I want to
add an IP address, I change the state and rerun 'deploy'. In the past
I used to run cfengine. Later I wrote cfruby/cfenjin which I still run
today for deployment and updates. For me it is time to write something
that plays well with GNU Guix.

Are you envisaging something like that? Something that reruns state
and updates? It is a lot more complicated because you need a way to
define state, modify files, allow for transaction and roll-back.
Ideally the system should execute in parallel (for speed) and be
ordered (i.e. if two methods change the same file the order matters).
BTW cfruby lacks transactions and parallel execution, we could start
without.

The first step is to define state by creating 'classes' of machines.
One class could be deploy 'sshd with IP restriction'. That would be a
good use case. 

Pj.

Re: Guix "ops"

[Thompson, David]

Hey Pjotr,

On Mon, Jun 1, 2015 at 11:18 AM, Pjotr Prins <pjotr.public12@thebird.nl> wrote:
>> There's also unanswered questions like: How should we keep track of
>> state?  How do we reconfigure already deployed machines?  How do we shut
>> down a deployment and unprovision the resources it used?  Basically, how
>> many hooks does the <platform> record type need to cover everything?
>
> The current 'deploy' basically generates a machine from scratch -
> there is some similar funtionality in Nix. I read the code and it is
> pretty simplistic for now which is possible when you generate a
> machine once.

Yes, the code is simplistic because I have only spent about a day or
two actually writing code for it. You'll notice that the <platform>
data type has many of its fields commented out.  Much work to do.

> What I would like to have is a state-machine that can be rerun within
> an installed deploy in the vein of Puppet/Chef/cfengine but without
> the complicated setup of a client/server model (the server 'model'
> should simply be a git repository with branches).

For persisting state between 'guix deploy' runs, I was going to write
an s-expression to a file.  The exact state written to such a file and
how it will be used will be defined by <platform> objects.  The state
could then be synced among all the workstations that are doing
deployments via whichever mechanism you prefer.  I would like to use
Git, too, but I don't 'guix deploy' to be concerned with such details.

> That means that after 'deploy' we can run the state and it
> checks/updates files that may have been changed. I use something like
> that to run my hosts.allow configuration (for example). When I want to
> add an IP address, I change the state and rerun 'deploy'. In the past
> I used to run cfengine. Later I wrote cfruby/cfenjin which I still run
> today for deployment and updates. For me it is time to write something
> that plays well with GNU Guix.
>
> Are you envisaging something like that? Something that reruns state
> and updates? It is a lot more complicated because you need a way to
> define state, modify files, allow for transaction and roll-back.
> Ideally the system should execute in parallel (for speed) and be
> ordered (i.e. if two methods change the same file the order matters).
> BTW cfruby lacks transactions and parallel execution, we could start
> without.

Sort of yes, sort of no.  What you are describing sounds quite
imperative.  In Guix, if we were to re-deploy a configuration to a
running remote system, we'd do the equivalent of what 'guix system
reconfigure' does now for local GuixSD machines: an atomic update of
the current system (changing the /run/current-system symlink).  'guix
deploy' cannot possibly do a good job of managing non-GuixSD systems
that just happen to run Guix.  I think it would be better to use the
existing configuration management tools for that.

> The first step is to define state by creating 'classes' of machines.
> One class could be deploy 'sshd with IP restriction'. That would be a
> good use case.

Are you proposing that we add a formal concept of "classes" in Guix or
is this just to illustrate an idea?  If the former, I think that pure
functions that return <operating-system> objects is far more powerful
than a primitive class labeling system.

Hope this helps clarify some things.  Thoughts?

- Dave

Guix deploy (and replace Puppet/Chef)

[Pjotr Prins]

(changed subject)

On Mon, Jun 01, 2015 at 12:49:31PM -0400, Thompson, David wrote:
> > Are you envisaging something like that? Something that reruns state
> > and updates? It is a lot more complicated because you need a way to
> > define state, modify files, allow for transaction and roll-back.
> > Ideally the system should execute in parallel (for speed) and be
> > ordered (i.e. if two methods change the same file the order matters).
> > BTW cfruby lacks transactions and parallel execution, we could start
> > without.
> 
> Sort of yes, sort of no.  What you are describing sounds quite
> imperative.  

Right.

> In Guix, if we were to re-deploy a configuration to a
> running remote system, we'd do the equivalent of what 'guix system
> reconfigure' does now for local GuixSD machines: an atomic update of
> the current system (changing the /run/current-system symlink).  'guix
> deploy' cannot possibly do a good job of managing non-GuixSD systems
> that just happen to run Guix.  I think it would be better to use the
> existing configuration management tools for that.

OK, this sounds exciting and could certainly work well. I guess
hosts.allow would be an input to an sshd builder, right, so different
configurations become their own subtrees in the store. I like that
idea. hosts.allow (as a complication) is actually part of tcp-wrappers
so it would have to be configured for all tools that it addresses on a
machine. I presume hosts.allow would be stored in the store too.

> > The first step is to define state by creating 'classes' of machines.
> > One class could be deploy 'sshd with IP restriction'. That would be a
> > good use case.
> 
> Are you proposing that we add a formal concept of "classes" in Guix or
> is this just to illustrate an idea?  If the former, I think that pure
> functions that return <operating-system> objects is far more powerful
> than a primitive class labeling system.
> 
> Hope this helps clarify some things.  Thoughts?

I am not too clear on the OS objects, but maybe I should play with
deploy first. I guess what you are saying is that a machine
configuration is generated by its s-expression(s) so we don't need
classes. Wherever these s-expressions come from is an implementation
issue. Right?

Pj.

Re: Guix "ops"

[Christopher Allan Webber]

[-- Attachment #1: Type: text/plain, Size: 1197 bytes --]

David Thompson writes:

> Hello again Carlos,
>
> Carlos Sosa <gnusosa@gnusosa.net> writes:
>
>>   I like the idea of 'guix deploy', and maybe something to propagates
>>   given configuration files, like 'guix config prepare' and later 'guix
>>   config apply'.
>>
>> Now, how can I contribute? work the guix command?
>>
>> Let me know if you have a specific repository or place to find any work
>> done on this.
>
> I have just pushed a new branch called "wip-deploy" to the official guix
> repository.  Since this branch is prefixed with "wip-", expect it to be
> rebased frequently.  There's not much code here yet, but a very simple
> prototype has been implemented that supports the creation of local QEMU
> VMs.
>
> To take it for a spin, add something like this to a file, let's call it
> "deployment.scm":

I've confirmed that the above works and works great.  I wanted to play
with it with current master, so I rebased the current branch on top of
it.  It's a mere single patch at the moment, so here's the patch with
appropriate conflicts resolved, in case anyone wants to play with it
with master (or in case David wants someone else to handle the conflict
resolving work for them ;))


[-- Attachment #2: 0001-scripts-Add-deploy.patch --]
[-- Type: text/x-diff, Size: 12684 bytes --]

From 25047d057c2adc30901b3052bf5017a6763741a1 Mon Sep 17 00:00:00 2001
From: David Thompson <dthompson2@worcester.edu>
Date: Mon, 13 Apr 2015 19:14:31 -0400
Subject: [PATCH] scripts: Add deploy.

* gnu/machines.scm: New file.
* gnu-system.am (GNU_SYSTEM_MODULES): Add it.
* guix/scripts/deploy.scm: New file.
* Makefile.am (MODULES): Add it.
* gnu.scm: Export (gnu machines) symbols.
* gnu/system/vm.scm (virtualized-operating-system): Export it.
---
 Makefile.am             |   1 +
 gnu-system.am           |   4 +-
 gnu.scm                 |   1 +
 gnu/machines.scm        | 125 +++++++++++++++++++++++++++++++++++++++
 gnu/system/vm.scm       |   2 +
 guix/scripts/deploy.scm | 153 ++++++++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 285 insertions(+), 1 deletion(-)
 create mode 100644 gnu/machines.scm
 create mode 100644 guix/scripts/deploy.scm

diff --git a/Makefile.am b/Makefile.am
index 7059a8f..9458b2c 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -121,6 +121,7 @@ MODULES =					\
   guix/scripts/publish.scm			\
   guix/scripts/edit.scm				\
   guix/scripts/size.scm				\
+  guix/scripts/deploy.scm			\
   guix.scm					\
   $(GNU_SYSTEM_MODULES)
 
diff --git a/gnu-system.am b/gnu-system.am
index d6369b5..d2d6f79 100644
--- a/gnu-system.am
+++ b/gnu-system.am
@@ -359,7 +359,9 @@ GNU_SYSTEM_MODULES =				\
   gnu/build/linux-container.scm			\
   gnu/build/linux-initrd.scm			\
   gnu/build/linux-modules.scm			\
-  gnu/build/vm.scm
+  gnu/build/vm.scm				\
+						\
+  gnu/machines.scm
 
 
 patchdir = $(guilemoduledir)/gnu/packages/patches
diff --git a/gnu.scm b/gnu.scm
index e3147b3..5cd1dea 100644
--- a/gnu.scm
+++ b/gnu.scm
@@ -42,6 +42,7 @@
         (gnu services base)
         (gnu packages)
         (gnu packages base)
+        (gnu machines)
         (guix gexp)))                             ; so gexps can be used
 
     (for-each (let ((i (module-public-interface (current-module))))
diff --git a/gnu/machines.scm b/gnu/machines.scm
new file mode 100644
index 0000000..2276732
--- /dev/null
+++ b/gnu/machines.scm
@@ -0,0 +1,125 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2015 David Thompson <davet@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu machines)
+  #:use-module (guix records)
+  #:use-module (gnu system)
+  #:use-module (gnu system vm)
+  #:use-module (ice-9 match)
+  #:use-module (srfi srfi-1)
+  #:export (deployment
+            make-deployment
+            deployment?
+            deployment-name
+            deployment-machines
+
+            machine
+            make-machine
+            machine?
+            machine-name
+            machine-system
+            machine-platform
+
+            platform
+            make-platform
+            platform-name
+            platform-description
+            platform-provision
+            platform-install
+            platform-reconfigure
+            platform-boot
+            platform-reboot
+            platform-halt
+            platform-destroy
+
+            machine-os-for-platform
+            provision-machine
+            boot-machine
+
+            local-vm))
+
+(define-record-type* <deployment> deployment
+  make-deployment
+  deployment?
+  (name deployment-name) ; string
+  (machines deployment-machines)) ; list of <machine>
+
+(define-record-type* <machine> machine
+  make-machine
+  machine?
+  (name machine-name) ; string
+  (system machine-system) ; <operating-system>
+  (platform machine-platform)) ; <platform>
+
+(define-record-type* <platform> platform
+  make-platform
+  platform?
+  (name platform-name) ; string
+  (description platform-description) ; string
+  (transform platform-transform) ; procedure
+  (provision platform-provision) ; procedure
+  ;; (install platform-install) ; procedure
+  ;; (reconfigure platform-reconfigure) ; procedure
+  (boot platform-boot) ; procedure
+  ;; (reboot platform-reboot) ; procedure
+  ;; (halt platform-halt) ; procedure
+  ;; (destroy platform-destroy) ; procedure
+  )
+
+(define (machine-os-for-platform machine)
+  ((platform-transform (machine-platform machine)) (machine-system machine)))
+
+(define (provision-machine machine)
+  (let ((os (machine-os-for-platform machine)))
+    ((platform-provision (machine-platform machine)) os)))
+
+(define (boot-machine machine state)
+  ((platform-boot (machine-platform machine)) state))
+
+(use-modules (guix monads)
+             (guix derivations)
+             (guix store)
+             (gnu services networking))
+
+(define* (local-vm #:key (ip-address "10.0.2.10"))
+  (platform
+   (name "local-vm")
+   (description "Local QEMU/KVM platform")
+   (transform
+    (lambda (os)
+      (let ((os (operating-system (inherit os)
+                  (services
+                   (cons
+                    (static-networking-service "eth0" ip-address
+                                               #:name-servers '("10.0.2.3")
+                                               #:gateway "10.0.2.2")
+                    (operating-system-user-services os))))))
+        (virtualized-operating-system os '()))))
+   (provision
+    (lambda (os)
+      (mlet %store-monad
+          ((vm-script (system-qemu-image/shared-store-script os)))
+        (mbegin %store-monad
+          (built-derivations (list vm-script))
+          (return (derivation-output-path
+                   (assoc-ref (derivation-outputs vm-script) "out")))))))
+   (boot
+    (lambda (script)
+      (match (primitive-fork)
+        (0 (primitive-exit (system* script)))
+        (pid #t))))))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 2520493..20f95d5 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -58,6 +58,8 @@
             qemu-image
             system-qemu-image
 
+            virtualized-operating-system
+
             system-qemu-image/shared-store
             system-qemu-image/shared-store-script
             system-disk-image))
diff --git a/guix/scripts/deploy.scm b/guix/scripts/deploy.scm
new file mode 100644
index 0000000..514d08a
--- /dev/null
+++ b/guix/scripts/deploy.scm
@@ -0,0 +1,153 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2015 David Thompson <davet@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts deploy)
+  #:use-module (guix ui)
+  #:use-module (guix store)
+  #:use-module (guix derivations)
+  #:use-module (guix packages)
+  #:use-module (guix profiles)
+  #:use-module (guix utils)
+  #:use-module (guix monads)
+  #:use-module (guix build utils)
+  #:use-module (guix scripts build)
+  #:use-module (gnu packages)
+  #:use-module (gnu system)
+  #:use-module (gnu system vm)
+  #:use-module (gnu machines)
+  #:use-module (ice-9 format)
+  #:use-module (ice-9 match)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-37)
+  #:use-module (srfi srfi-98)
+  #:export (guix-deploy))
+
+(define (show-help)
+  (display (_ "Usage: guix deploy [OPTION] ACTION FILE
+Manage your data beans without disturbing Terry the data goblin.\n"))
+  (newline)
+  (display (_ "The valid values for ACTION are:\n"))
+  (display (_ "\
+  - 'build', build all of the operating systems without deploying\n"))
+  (display (_ "\
+  - 'init', provision and install the operating systems\n"))
+  (display (_ "\
+  - 'reconfigure', update an existing deployment\n"))
+  (display (_ "\
+  - 'destroy', unprovision the deployed operating systems\n"))
+  (display (_ "
+  -e, --expression=EXPR  create environment for the package that EXPR
+                         evaluates to"))
+  (newline)
+  (show-build-options-help)
+  (newline)
+  (display (_ "
+  -h, --help             display this help and exit"))
+  (display (_ "
+  -V, --version          display version information and exit"))
+  (newline)
+  (show-bug-report-information))
+
+(define %default-options
+  `((substitutes? . #t)
+    (max-silent-time . 3600)
+    (verbosity . 0)))
+
+(define %options
+  (cons* (option '(#\h "help") #f #f
+                 (lambda args
+                   (show-help)
+                   (exit 0)))
+         (option '(#\V "version") #f #f
+                 (lambda args
+                   (show-version-and-exit "guix deploy")))
+         %standard-build-options))
+
+(define-syntax-rule (return* body ...)
+  "Generate the monadic form of BODY, an expression evaluated for its
+side-effects.  The result is always #t."
+  (return (begin body ... #t)))
+
+(define (deployment-derivations deployment)
+  (map (lambda (machine)
+         (operating-system-derivation
+          (machine-os-for-platform machine)))
+       (deployment-machines deployment)))
+
+(define (build-deployment deployment)
+  (mlet* %store-monad
+      ((drvs (sequence %store-monad (deployment-derivations deployment))))
+    (mbegin %store-monad
+      (show-what-to-build* drvs)
+      (built-derivations drvs)
+      (return*
+       (for-each (lambda (drv)
+                   (display (derivation->output-path drv))
+                   (newline))
+                 drvs)))))
+
+(define (provision-deployment deployment)
+  (sequence %store-monad
+            (map (lambda (machine)
+                   (mlet %store-monad
+                       ((state (provision-machine machine)))
+                     (return (list machine state))))
+                 (deployment-machines deployment))))
+
+(define (spawn-deployment deployment)
+  (mlet %store-monad
+      ((states (provision-deployment deployment)))
+    (sequence %store-monad
+              (map (match-lambda
+                    ((machine state)
+                     (return* (boot-machine machine state))))
+                   states))))
+
+(define (perform-action action deployment)
+  (case action
+    ((build) (build-deployment deployment))
+    ((provision) (provision-deployment deployment))
+    ((spawn) (spawn-deployment deployment))))
+
+(define (guix-deploy . args)
+  (define (parse-sub-command-or-config arg result)
+    (cond
+     ((assoc-ref result 'config)
+      (leave (_ "~a: extraneous argument~%") arg))
+     ((assoc-ref result 'action)
+      (alist-cons 'config arg result))
+     (else
+      (let ((action (string->symbol arg)))
+        (case action
+          ((build provision spawn)
+           (alist-cons 'action action result))
+          (else (leave (_ "~a: unknown action~%") action)))))))
+
+  (with-error-handling
+    (let* ((opts (args-fold* args %options
+                             (lambda (opt name arg result)
+                               (leave (_ "~A: unrecognized option~%") name))
+                             parse-sub-command-or-config %default-options))
+           (action (assoc-ref opts 'action))
+           (deployment (load (assoc-ref opts 'config))))
+      (with-store store
+        (run-with-store store
+          (mbegin %store-monad
+            (set-build-options-from-command-line* opts)
+            (perform-action action deployment)))))))
-- 
2.1.4

Re: Guix "ops"

[Christopher Allan Webber]

David Thompson writes:

> Hello again Carlos,
>
> Carlos Sosa <gnusosa@gnusosa.net> writes:
>
>>   I like the idea of 'guix deploy', and maybe something to propagates
>>   given configuration files, like 'guix config prepare' and later 'guix
>>   config apply'.
>>
>> Now, how can I contribute? work the guix command?
>>
>> Let me know if you have a specific repository or place to find any work
>> done on this.
>
> I have just pushed a new branch called "wip-deploy" to the official guix
> repository.  Since this branch is prefixed with "wip-", expect it to be
> rebased frequently.  There's not much code here yet, but a very simple
> prototype has been implemented that supports the creation of local QEMU
> VMs.

Just FYI, I've rebased this branch and pushed to origin.

There's a lot of good ideas in this thread.  Will be good to make
progress on them!

Re: Guix "ops"

[Ludovic Courtès]

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> David Thompson writes:
>
>> Hello again Carlos,
>>
>> Carlos Sosa <gnusosa@gnusosa.net> writes:
>>
>>>   I like the idea of 'guix deploy', and maybe something to propagates
>>>   given configuration files, like 'guix config prepare' and later 'guix
>>>   config apply'.
>>>
>>> Now, how can I contribute? work the guix command?
>>>
>>> Let me know if you have a specific repository or place to find any work
>>> done on this.
>>
>> I have just pushed a new branch called "wip-deploy" to the official guix
>> repository.  Since this branch is prefixed with "wip-", expect it to be
>> rebased frequently.  There's not much code here yet, but a very simple
>> prototype has been implemented that supports the creation of local QEMU
>> VMs.
>
> Just FYI, I've rebased this branch and pushed to origin.

Thanks!

> There's a lot of good ideas in this thread.  Will be good to make
> progress on them!

Yup!  The need for this tool is becoming stronger, especially now that
we’re starting to run GuixSD on our build farm machines.

Ludo’.

Re: Guix "ops"

[Christopher Allan Webber]

Ludovic Courtès writes:

>> There's a lot of good ideas in this thread.  Will be good to make
>> progress on them!
>
> Yup!  The need for this tool is becoming stronger, especially now that
> we’re starting to run GuixSD on our build farm machines.
>
> Ludo’.

So yeah, I'm going to start playing around with building some on some of
these ideas soonish.  I could use some advice, though.  Assume I'm able
to build the right scheme representation of the system I want to be run
remotely on another machine (whether it's a gexp, whatever).  What's the
right way to go about this?  Assume for the moment that I have a remote
"target" machine already running GuixSD and I can ssh into it.

 - Should I write out an sexp of the system I want to be built, write it
   to a scheme file I copy over, and then invoke the guile command line
   tools to start it up?

 - Should I write out an sexp of the script I want to run and execute it
   as a normal guile program?

 - Should I build the entire derivation of the system I want to run on
   the remote machine locally first, then copy that over?  (I assume
   this is possible, and eventually desirable, especially if doing
   mass deployments?  But it might not be desirable in every case.)
   Would that use the substitute mechanism?

Thoughts and guidance appreciated!
 - Chris

Re: Guix "ops"

[Ludovic Courtès]

Howdy!

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> So yeah, I'm going to start playing around with building some on some of
> these ideas soonish.  I could use some advice, though.  Assume I'm able
> to build the right scheme representation of the system I want to be run
> remotely on another machine (whether it's a gexp, whatever).  What's the
> right way to go about this?  Assume for the moment that I have a remote
> "target" machine already running GuixSD and I can ssh into it.
>
>  - Should I write out an sexp of the system I want to be built, write it
>    to a scheme file I copy over, and then invoke the guile command line
>    tools to start it up?

No!

>  - Should I write out an sexp of the script I want to run and execute it
>    as a normal guile program?

No!

>  - Should I build the entire derivation of the system I want to run on
>    the remote machine locally first, then copy that over?  (I assume
>    this is possible, and eventually desirable, especially if doing
>    mass deployments?  But it might not be desirable in every case.)
>    Would that use the substitute mechanism?

Yes!  :-)

Essentially deployment would work like this:

  1. Compute the system derivation and build it locally (i.e., on the
     machine where ‘guix deploy’ is running.)

     As a user, you can choose to have offloading setup such that the
     actual build will take place on (some of) the target machines, but
     that’s completely orthogonal.

  2. Send the derivation out to the target(s) that are real machines.
     For targets that are local containers or VMs, there’s nothing to
     do.

  3. On targets that are real machines, perform the equivalent of ‘guix
     system reconfigure’—i.e., update the /run/current-system symlink,
     restart Shepherd services that can be restarted, etc.

IIRC David was testing using VMs and containers as the targets (the
<platform> record¹) because it’s easier.

Does that make sense?

Hopefully David will correct me.  :-)

Ludo’.

¹ https://lists.gnu.org/archive/html/guix-devel/2015-07/msg00320.html

Re: Guix "ops"

[Christopher Allan Webber]

Ludovic Courtès writes:

> Christopher Allan Webber <cwebber@dustycloud.org> skribis:
>
>>  - Should I build the entire derivation of the system I want to run on
>>    the remote machine locally first, then copy that over?  (I assume
>>    this is possible, and eventually desirable, especially if doing
>>    mass deployments?  But it might not be desirable in every case.)
>>    Would that use the substitute mechanism?
>
> Yes!  :-)
>
> Essentially deployment would work like this:
>
>   1. Compute the system derivation and build it locally (i.e., on the
>      machine where ‘guix deploy’ is running.)
>
>      As a user, you can choose to have offloading setup such that the
>      actual build will take place on (some of) the target machines, but
>      that’s completely orthogonal.
>
>   2. Send the derivation out to the target(s) that are real machines.
>      For targets that are local containers or VMs, there’s nothing to
>      do.
>
>   3. On targets that are real machines, perform the equivalent of ‘guix
>      system reconfigure’—i.e., update the /run/current-system symlink,
>      restart Shepherd services that can be restarted, etc.
>
> IIRC David was testing using VMs and containers as the targets (the
> <platform> record¹) because it’s easier.
>
> Does that make sense?
>
> Hopefully David will correct me.  :-)

Ok, it does make sense!

It's nice to see the examples in the docs of exporting a system over
ssh, even.  Anyway, I played with "guix archive" this morning; it works
well.  So, sending an entire closure over the network: should be easy.

I see that there's a --missing field; I'm a little bit unsure of how two
machines would coordinate here though with the existing tooling... it
doesn't look like we have a way to export the list of packges that
--missing could then read in?  And then you'd need to feed whatever
--missing gave you back into --export I guess.

Anyway, it does seem like this is the right direction!  Exciting!
 - Chris

>
> Ludo’.
>
> ¹ https://lists.gnu.org/archive/html/guix-devel/2015-07/msg00320.html

Re: Guix "ops"

[Ludovic Courtès]

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> Ludovic Courtès writes:
>
>> Christopher Allan Webber <cwebber@dustycloud.org> skribis:
>>
>>>  - Should I build the entire derivation of the system I want to run on
>>>    the remote machine locally first, then copy that over?  (I assume
>>>    this is possible, and eventually desirable, especially if doing
>>>    mass deployments?  But it might not be desirable in every case.)
>>>    Would that use the substitute mechanism?
>>
>> Yes!  :-)
>>
>> Essentially deployment would work like this:
>>
>>   1. Compute the system derivation and build it locally (i.e., on the
>>      machine where ‘guix deploy’ is running.)
>>
>>      As a user, you can choose to have offloading setup such that the
>>      actual build will take place on (some of) the target machines, but
>>      that’s completely orthogonal.
>>
>>   2. Send the derivation out to the target(s) that are real machines.
>>      For targets that are local containers or VMs, there’s nothing to
>>      do.
>>
>>   3. On targets that are real machines, perform the equivalent of ‘guix
>>      system reconfigure’—i.e., update the /run/current-system symlink,
>>      restart Shepherd services that can be restarted, etc.
>>
>> IIRC David was testing using VMs and containers as the targets (the
>> <platform> record¹) because it’s easier.
>>
>> Does that make sense?
>>
>> Hopefully David will correct me.  :-)
>
> Ok, it does make sense!
>
> It's nice to see the examples in the docs of exporting a system over
> ssh, even.  Anyway, I played with "guix archive" this morning; it works
> well.  So, sending an entire closure over the network: should be easy.
>
> I see that there's a --missing field; I'm a little bit unsure of how two
> machines would coordinate here though with the existing tooling... it
> doesn't look like we have a way to export the list of packges that
> --missing could then read in?  And then you'd need to feed whatever
> --missing gave you back into --export I guess.

Yes, there’s an example of that in (guix scripts offload), in
‘send-files’.

Essentially you do:

  guix archive --export \
    `guix gc -R the-thing-to-send | ssh host guix archive --missing` | \
    ssh host guix archive --import

HTH!

Ludo’.

It's time to build "guix deploy"

[Christopher Lemmer Webber]

Hi,

This has come up several times.  A lot of us want "guix deploy".
Personally, I'm running a variety of Debian servers and one Guix server
and I am terrible at maintaining all of them.

It's time for Guix Deploy... or it's time for me to give up and use
something like Ansible + Debian.  (Egads!  I don't want to do that.)

David's thoughts on this are below, and here's the original thread:

Original thread can be found at the links below:
  https://lists.gnu.org/archive/html/guix-devel/2015-04/msg00525.html
  https://lists.gnu.org/archive/html/guix-devel/2015-05/msg00007.html
  https://lists.gnu.org/archive/html/guix-devel/2015-06/msg00006.html

There is a heavily, heavily bitrotted branch named "wip-deploy" where
David originally started exploring these ideas.  Conveniently, it's all
in one commit:

  https://git.savannah.gnu.org/cgit/guix.git/commit/?h=wip-deploy&id=fcd6fc84e493d05be1f7590ee77509c81ac315c2

That seems like a good starting point.  But I know David feels that with
real-world experience in devops type work that actually things would be
a bit different than what's described in his original email.  I'm not
sure myself what would be different.  It would be helpful to hear Dave
weigh in on that point.

Maybe Dave and I can meet up IRL now that we're close enough to each
other to chat about it.  But I know it's less fun than it used to be for
Dave to consider this because now that's Dave's actual job... but all
the more reason we need Dave's wisdom! :)


David Thompson writes:

> Hey folks,
>
> I'm writing this in an attempt to "think out loud" about a deployment
> automation tool for GuixSD.
>
> Guix needs a NixOps[0] equivalent.  NixOps is the Nix project's
> deployment automation tool.  I read the source code[1] a bit, and here's
> what I've learned about it:
>
> * The central data type is a "deployment", which is a Nix expression
>   consisting of one or more named OS configs
>
> * The OS configs contain extra data that specifies the target platform:
>   VirtualBox, EC2, NixOS container, etc.  Each platform implements the
>   generic MachineDefinition and MachineState interfaces.
>
> * The 'nixops' command is stateful.  It stores necessary state about
>   deployed systems in a special directory ($HOME/.nixops by default),
>   such as the IP addresses of the deployed systems.  Because of this,
>   each deployment needs a unique identifier.
>
> * Deployments may be parameterized by values known only at deploy time.
>   This covers cases such as a web application server needing to know the
>   IP address of the database server.
>
> * There are useful subcommands to check the status of each system or ssh
>   into one of them.
>
> Here's a rough outline of how I'm thinking of implementing similar
> features in Guix:
>
> * Introduce new data types:
>
>   * <platform>: The generic interface type for implementing deployment
>     targets.  Its fields hold procedures for various actions like
>     'provision', 'destroy', 'boot', or 'reboot'.  I haven't determined
>     the precise list of actions needed, but it will become apparent as
>     platforms are added.
>
>   * <machine>: Describes a single system in the deployment.  Contains a
>     name string, an <operating-system>, and a <platform>.
>
>   * <deployment>: Contains a name string and a list of <machine> and
>     procedures.  Procedures in the list should accept an argument
>     containing the deployment results of the non-parameterized machines
>     and return a <machine>.
>
> * Use a simple s-exp deployment state format.  Store the state in
>   $HOME/.config/guix by default.
>
> * Implement a 'guix ops' subcommand roughly the same actions as NixOps:
>   create, deploy, start, stop, destroy, list, info, check, ssh, etc.
>
> * The bulk of the work will be creating <platform> objects that actually
>   provision various types of resources.  For prototyping, a
>   'local-vm-platform' would be enough to test that the whole system
>   works.
>
> Anyone want to join in on this brainstorming party?  Your thoughts are
> appreciated!
>
> [0] http://nixos.org/nixops/
> [1] https://github.com/NixOS/nixops

Re: It's time to build "guix deploy"

[Pjotr Prins]

I am also interested in 'guix deploy', especially if it can run on top
of Debian and handle HOME directories. Currently using a mixture of
sysadmin heroics and my own cfengine style Ruby deploy tool ;). Happy
to switch, test and hack. But not enough time to run with it.

Pj.

Re: It's time to build "guix deploy"

[Christopher Lemmer Webber]

Pjotr Prins writes:

> I am also interested in 'guix deploy', especially if it can run on top
> of Debian and handle HOME directories. Currently using a mixture of
> sysadmin heroics and my own cfengine style Ruby deploy tool ;). Happy
> to switch, test and hack. But not enough time to run with it.
>
> Pj.

I am trying to think, "what can I do to help advance things further",
and maybe my best next step is to write minimalist user stories with
mockups of configurations and deployments based off of davexunit's old
branch?  That way I can think through what's there and what I need yet.

 - Chris

Re: It's time to build "guix deploy"

[Amirouche Boubekki]

On 2019-02-11 15:47, Christopher Lemmer Webber wrote:
> Pjotr Prins writes:
> 
>> I am also interested in 'guix deploy', especially if it can run on top
>> of Debian and handle HOME directories. Currently using a mixture of
>> sysadmin heroics and my own cfengine style Ruby deploy tool ;). Happy
>> to switch, test and hack. But not enough time to run with it.
>> 
>> Pj.
> 
> I am trying to think, "what can I do to help advance things further",
> and maybe my best next step is to write minimalist user stories with
> mockups of configurations and deployments based off of davexunit's old
> branch?  That way I can think through what's there and what I need yet.
> 
>  - Chris


Great idea!

Re: It's time to build "guix deploy"

[Christopher Lemmer Webber]

Pjotr Prins writes:

> I am also interested in 'guix deploy', especially if it can run on top
> of Debian and handle HOME directories.

I should also say: I'm not considering how "guix deploy" could or would
handle running on top of Debian *or* handle HOME directories.  Both of
those feel out of scope to me, but for different reasons:

 - "guix deploy" would deploy by, well, building guix system profiles
   and then pushing them out.  So I'm not sure how Debian would fit in
   there...
 - Mutable state of the system is hard to tackle.  We might need to
   tackle some degree of it, but I think that would be in
   post-install-hooks type things which might sometimes do that
   on server?

Re: It's time to build "guix deploy"

[Pjotr Prins]

On Mon, Feb 11, 2019 at 09:57:59AM -0500, Christopher Lemmer Webber wrote:
> Pjotr Prins writes:
> 
> > I am also interested in 'guix deploy', especially if it can run on top
> > of Debian and handle HOME directories.
> 
> I should also say: I'm not considering how "guix deploy" could or would
> handle running on top of Debian *or* handle HOME directories.  Both of
> those feel out of scope to me, but for different reasons:
> 
>  - "guix deploy" would deploy by, well, building guix system profiles
>    and then pushing them out.  So I'm not sure how Debian would fit in
>    there...
>  - Mutable state of the system is hard to tackle.  We might need to
>    tackle some degree of it, but I think that would be in
>    post-install-hooks type things which might sometimes do that
>    on server?

Sure. There was a message recently about having Guix doing
'outer' state too. I lost what thread it was in. I am interested not
least because I need a transition path to GuixSD. If we were on VMs it
would be relatively easy, but our servers are on bare metal.

Pj.

Re: It's time to build "guix deploy"

[Thompson, David]

Hi Chris,

Here we go again, eh? :)

On Mon, Feb 11, 2019 at 8:31 AM Christopher Lemmer Webber
<cwebber@dustycloud.org> wrote:
>
> Hi,
>
> This has come up several times.  A lot of us want "guix deploy".
> Personally, I'm running a variety of Debian servers and one Guix server
> and I am terrible at maintaining all of them.

I have just a single Linode VPS that I can't be bothered to maintain
most of the time. I would like to switch to Guix, as well.

> It's time for Guix Deploy... or it's time for me to give up and use
> something like Ansible + Debian.  (Egads!  I don't want to do that.)
>
> David's thoughts on this are below, and here's the original thread:
>
> Original thread can be found at the links below:
>   https://lists.gnu.org/archive/html/guix-devel/2015-04/msg00525.html

Wow, 2015. I was so young and full of hope then. ;)

>   https://lists.gnu.org/archive/html/guix-devel/2015-05/msg00007.html
>   https://lists.gnu.org/archive/html/guix-devel/2015-06/msg00006.html
>
> There is a heavily, heavily bitrotted branch named "wip-deploy" where
> David originally started exploring these ideas.  Conveniently, it's all
> in one commit:
>
>   https://git.savannah.gnu.org/cgit/guix.git/commit/?h=wip-deploy&id=fcd6fc84e493d05be1f7590ee77509c81ac315c2

Useful for context, but the code can probably be tossed at this point.

> That seems like a good starting point.  But I know David feels that with
> real-world experience in devops type work that actually things would be
> a bit different than what's described in his original email.  I'm not
> sure myself what would be different.  It would be helpful to hear Dave
> weigh in on that point.

Sure, since 2015 I've become the lead devops person at my company, so
I like to think I'm a bit wiser now.

> Maybe Dave and I can meet up IRL now that we're close enough to each
> other to chat about it.  But I know it's less fun than it used to be for
> Dave to consider this because now that's Dave's actual job... but all
> the more reason we need Dave's wisdom! :)

We could meet up IRL about this and I can try to make an earnest
effort to deal with this. I think what has stopped me in the past is
the sheer size of this project, and maybe dramatically scaling down
the scope will allow us to get *something* out the door.  Here are
some general use-cases I know about for deployments, roughly ordered
from small scale to large scale, and least complex to most complex:

* Managing a physical machine or two that have been given memorable
names that you update in-place (home scale)
* Managing a virtual machine or two that have been given memorable
names that you update in-place (blog scale)
* Managing a large number of virtual machines whose names don't matter
that you update in-place (proto-cloud scale)
* Managing a large number of virtual machines whose names don't matter
that are replaced when there is an update (cloud scale)
* Managing 1 or more clusters of physical machines (datacenter scale)
* Managing 1 or more clusters of physical machines and virtual
machines ("corporation with a datacenter that is moving some stuff to
the cloud" scale)

There are, of course, more scenarios to consider (haven't even touched
upon things like a Kubernetes cluster), but this is enough to
illustrate the point that is a great diversity in setups.  How many
machines are there? Are the bare metal, virtual machines, or a mix of
both? In the case of virtual machines, are updates applied in an
immutable fashion or not?  If immutable, which technique (blue-green,
rolling release, etc.)?  It makes my head spin to think about all the
use-cases.

So... let's start small. Can we write a tool that handles in-place
updates to machines (physical or virtual) whose name and IP address we
know well (our special pet servers) without precluding the possibility
of scaling up to more sophisticated architectures?  This would address
the "home" and "blog" scale items above, which is probably what most
of the people actually using Guix today would want.  I got stuck
trying to do in-place updates to remote machines years ago, but that
was before Ludo made it easy to connect to remote systems.

Other thoughts?

Yours in vaporware,

- Dave

Re: It's time to build "guix deploy"

[Ricardo Wurmus]

Thompson, David <dthompson2@worcester.edu> writes:

> Other thoughts?

Just for reference: to update Berlin build nodes I use this script:

    https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/install-berlin.scm

It’s not great, but it’s been helpful.

Berlin consists of a head node and many almost identical servers.  To
update one or more servers I run the script on the head node, which
generates operating system configuration variants for each of the
requested servers, builds the systems (offloading to all of the
connected build nodes), copies the system closures to the target
systems, and then runs “reconfigure” on the targets.

Since the operating system configuration record cannot be serialized,
the build nodes need to have a copy of the code that’s used to generate
the operating system configuration.  Not great.  (They only need it to
run “reconfigure”; they wouldn’t need that if “reconfigure” could
operate remotely.)

Anyway, I thought I’d share this with y’all.

--
Ricardo

Re: It's time to build "guix deploy"

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 2342 bytes --]

Hi devel!

I'm *very* interested in this development... actually remote
provisioning of operating system - and services configuration - is the
reason I'm here :-)

"guix deploy infrastructure.scm" it's everyone dream, no? :-O

unfortunately I'm still not able to help in this development, I'm
writing just to show my *great* appreciation for each of you that *will*
make this dream come true

maybe a *design* document stored somewhere collecting all past and
present thoughts on this matter would help attracting contributions

Ricardo Wurmus <rekado@elephly.net> writes:

> Thompson, David <dthompson2@worcester.edu> writes:
>
>> Other thoughts?
>
> Just for reference: to update Berlin build nodes I use this script:
>
>     https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/install-berlin.scm
>
> It’s not great, but it’s been helpful.

thanks for sharing! (even if I can still barely understand what your
script does)

actually mainenance.git is full of treasures :-)

> Berlin consists of a head node and many almost identical servers.

AFAIU remote servers could be completely different each other for your
script to do its job, or am I missing something?

> To
> update one or more servers I run the script on the head node, which
> generates operating system configuration variants for each of the
> requested servers, builds the systems (offloading to all of the
> connected build nodes), copies the system closures to the target
> systems, and then runs “reconfigure” on the targets.

explained this way seems easy :-O

> Since the operating system configuration record cannot be serialized,

is there any plan or wip on this kind of serialization?

> the build nodes need to have a copy of the code that’s used to generate
> the operating system configuration.  Not great.  (They only need it to
> run “reconfigure”; they wouldn’t need that if “reconfigure” could
> operate remotely.)

"just" having a "guix system reconfigure --host <remote-hostname/IP>"
would be a *huge* feature

> Anyway, I thought I’d share this with y’all.

IMHO your remote host configuration technique deserves a dedicated blog
article... but I've already asked too much :-)

Thanks!

Giovanni

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: It's time to build "guix deploy"

[swedebugia]

[-- Attachment #1: Type: text/plain, Size: 2425 bytes --]

Hi :)

ons 2019-02-13 klockan 20:04 +0100 skrev Giovanni Biscuolo 
<g@xelera.eu>:
> 
> 
> Ricardo Wurmus <rekado@elephly.net> writes:
> 
>>  Thompson, David <dthompson2@worcester.edu> writes:
>> 
>>>  Other thoughts?
>> 
>>  Just for reference: to update Berlin build nodes I use this script:
>> 
>>      
>> https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/install-berlin.scm
>> 
>>  It’s not great, but it’s been helpful.
> 
> thanks for sharing! (even if I can still barely understand what your
> script does)

I understand most parts of it ;)
It is a real beauty and a testiment to the power of Guix and Guile.


> 
> actually mainenance.git is full of treasures :-)
> 
>>  Berlin consists of a head node and many almost identical servers.
> 
> AFAIU remote servers could be completely different each other for your
> script to do its job, or am I missing something?

Besides the host-ips being hardcoded and the
(berlin-build-machine-os) procedure from
(sysadmin build-machines) and the keys from
(sysadmin people) it seems pretty generic.

Ricardo, is (define (reconfigure-remote id guix-directory) dead code 
that could be commented out/removed?

> 
>>  To
>>  update one or more servers I run the script on the head node, which
>>  generates operating system configuration variants for each of the
>>  requested servers, builds the systems (offloading to all of the
>>  connected build nodes), copies the system closures to the target
>>  systems, and then runs “reconfigure” on the targets.
> 
> explained this way seems easy :-O
> 
>>  Since the operating system configuration record cannot be 
>> serialized,
> 
> is there any plan or wip on this kind of serialization?
> 
>>  the build nodes need to have a copy of the code that’s used to 
>> generate
>>  the operating system configuration.  Not great.  (They only need it 
>> to
>>  run “reconfigure”; they wouldn’t need that if 
>> “reconfigure” could
>>  operate remotely.)
> 
> "just" having a "guix system reconfigure --host <remote-hostname/IP>"
> would be a *huge* feature

Agreed, but we would need to supply both keys and generic-config to 
this command as well.

> 
>>  Anyway, I thought I’d share this with y’all.
> 
> IMHO your remote host configuration technique deserves a dedicated 
> blog
> article... but I've already asked too much :-)

Good idea!

[-- Attachment #2: Type: text/html, Size: 4058 bytes --]

Re: It's time to build "guix deploy"

[Pjotr Prins]

On Thu, Feb 14, 2019 at 08:14:15AM +0100, swedebugia wrote:
>    I understand most parts of it ;)
>    It is a real beauty and a testiment to the power of Guix and Guile.
> 
>    actually mainenance.git is full of treasures :-)
> 
>      Berlin consists of a head node and many almost identical servers.
> 
>    AFAIU remote servers could be completely different each other for your
>    script to do its job, or am I missing something?

So, essentially, the tools login via ssh and control guix remotely and
copy files which will also work on an underlying Debian. 

This looks a lot like the Python automation system I used in the past
with cloudbiolinux:

  https://github.com/pjotrp/cloudbiolinux

Later I created a simple Ruby system that takes YAML files as input and runs
*locally* . A remote invocation called that local system. The advantage
is that is does not lean on ssh too heavily (one login required and
you could move to a different client-server protocol easily - even
over http). I am still using that setup today, to configure web, mail
servers and home directory. The tool is here

  https://github.com/pjotrp/deploy

An example of use for emacs is emacs.yaml:

---
- copy-file:
    emacs:
      dest: .emacs
      mode: "400"
- dir:
    .emacs.d:
      source: emacs.d
      recursive: true

and the emacs files sit in a git directory in the same tree and get
copied across running 'deploy emacs.yaml'. It is not fancy, but it
works well. Of course we should not use YAML with guile ;). WISP would
work fine.

Note that I have used Cfengine extensively (even wrote my own clone),
Also used Chef and Puppet. 

After all that I ended up with writing a *simple* system that does not
keep track of state but simply copies files. Guix would do this better
by providing transactions and isolation. 

I think my preferred way to do this is to copy files into the store
and use guix deploy so symlink them from $HOME and /etc - i.e., a
profile with another layer of symlinks. This means that when a
profile goes out of scope the symlinks stop working too. They dangle,
that is all, but should be easy to harvest since you know what
directory you are linking in. 

Guix deploy runs locally on a machine and can be invoked
remotely. One advantage of running locally is that it is fast and much
easier to test.

That would work on Debian+Guix too.

Pj.

PS I like the logo :)

https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/berlin.scm#n50

(though not used there).  

Re: It's time to build "guix deploy"

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 1970 bytes --]

Hello Pjotr,

thanks for sharing your thougts!

Pjotr Prins <pjotr.public12@thebird.nl> writes:

[...]

> I am still using that setup today, to configure web, mail
> servers and home directory. The tool is here
>
>   https://github.com/pjotrp/deploy

quiting from the README: "`deploy’ is a deployment tool which operates
in the same domain as Chef, Puppet, Cfengine"

I'd call them "configuration management tools" instead of "deployment
tools", even if someway (*badly*) they can also deploy software and
apply the configuration

why not using (or defining [1]) system services in a Guix
operating-system declaration instead of *any* other configuration
management software?... except for legacy reasons

the very reason I'm here is I don't want to use *anymore* *any* of them,
with all due _respect_ for the venerable projects, your included!

I've used Puppet and some Ansible, studied CFengine and Salt
Stack... then discovered Nix and rigth next Guix: what else? :-)

[...]

> and the emacs files sit in a git directory in the same tree and get
> copied across running 'deploy emacs.yaml'.

yes, we still miss "stateless user services config" (Pierre Neidhardt
wrote an interesting summary here
https://lists.gnu.org/archive/html/guix-devel/2019-02/msg00128.html)

I'd like to be able to declaratively manage a *stateless* .config/
instead of managing configuration with dotfiles [2]

anyway Guix is _perfect_ to declare and deploy system services, what we
miss is a little more abstraction (from operating-system to
infrastructure?) and remote control of "guix system reconfigure"

am I missing something?

Thanks
Giovanni


[1] https://www.gnu.org/software/guix/manual/en/html_node/Defining-Services.html#Defining-Services

[2] I'm using myrepos with "stowable = true" for my dotfolders... but
I'll _never_ use something like it's Drupal extension 

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: It's time to build "guix deploy"

[Pjotr Prins]

On Thu, Feb 14, 2019 at 04:35:47PM +0100, Giovanni Biscuolo wrote:
> the very reason I'm here is I don't want to use *anymore* *any* of them,
> with all due _respect_ for the venerable projects, your included!

I did not intend to push my 'solution'. I want to get away from that
too!

> I've used Puppet and some Ansible, studied CFengine and Salt
> Stack... then discovered Nix and rigth next Guix: what else? :-)
> 
> [...]
> 
> > and the emacs files sit in a git directory in the same tree and get
> > copied across running 'deploy emacs.yaml'.
> 
> yes, we still miss "stateless user services config" (Pierre Neidhardt
> wrote an interesting summary here
> https://lists.gnu.org/archive/html/guix-devel/2019-02/msg00128.html)
> 
> I'd like to be able to declaratively manage a *stateless* .config/
> instead of managing configuration with dotfiles [2]
> 
> anyway Guix is _perfect_ to declare and deploy system services, what we
> miss is a little more abstraction (from operating-system to
> infrastructure?) and remote control of "guix system reconfigure"
> 
> am I missing something?

I think you misread my E-mail. No doubt my fault ;).

Pj.

Re: It's time to build "guix deploy"

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 523 bytes --]

Hi swedebugia!

swedebugia <swedebugia@riseup.net> writes:

[...]

> I understand most parts of it ;)
> It is a real beauty and a testiment to the power of Guix and Guile.

[...]

>> IMHO your remote host configuration technique deserves a dedicated 
>> blog
>> article... but I've already asked too much :-)
>
> Good idea!

given you understand most part of it, why don't you try to write an
exegesis of that code yourself ;-)

Thanks!
Giovanni

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: It's time to build "guix deploy"

[swedebugia]

Giovanni Biscuolo <g@xelera.eu> skrev: (14 februari 2019 15:17:34 CET)
>Hi swedebugia!
>
>swedebugia <swedebugia@riseup.net> writes:
>
>[...]
>
>> I understand most parts of it ;)
>> It is a real beauty and a testiment to the power of Guix and Guile.
>
>[...]
>
>>> IMHO your remote host configuration technique deserves a dedicated 
>>> blog
>>> article... but I've already asked too much :-)
>>
>> Good idea!
>
>given you understand most part of it, why don't you try to write an
>exegesis of that code yourself ;-)

I could certainly try to do that but Ricardo said himself "its not great..." so I don't know if it is a good idea.

Anyway it draws on quite a lot of modules so I would have to look at them too to get the whole picture. 

Which parts would you like explained? 

Hi
-- 
Sent from my k-9 mail for Android.

Re: It's time to build "guix deploy"

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 1025 bytes --]

Hi swedebugia,

swedebugia <swedebugia@riseup.net> writes:

[...]

>>given you understand most part of it, why don't you try to write an
>>exegesis of that code yourself ;-)

> I could certainly try to do that but Ricardo said himself "its not
> great..." so I don't know if it is a good idea.

yes, probably the code would need to be refactored/abstracted to be
useful "out of the box" for others

> Anyway it draws on quite a lot of modules so I would have to look at
> them too to get the whole picture.
>
> Which parts would you like explained?

I'd need a tutorial like
https://guix.info/blog/2018/a-packaging-tutorial-for-guix/
...something titled "manage a fleet of hosts with Guix" :-O

...but given the actual status of the code I realize it's not feasible,
so nevermind please :-)

I'll do my homework to learn enough Guile to understand that code on my
own and if I'll be able to build some abstraction on it

Thanks!
Giovanni

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: It's time to build "guix deploy"

[Christopher Lemmer Webber]

Thompson, David writes:

> Hi Chris,
>
> Here we go again, eh? :)
>
>>   https://lists.gnu.org/archive/html/guix-devel/2015-05/msg00007.html
>>   https://lists.gnu.org/archive/html/guix-devel/2015-06/msg00006.html
>>
>> There is a heavily, heavily bitrotted branch named "wip-deploy" where
>> David originally started exploring these ideas.  Conveniently, it's all
>> in one commit:
>>
>>   https://git.savannah.gnu.org/cgit/guix.git/commit/?h=wip-deploy&id=fcd6fc84e493d05be1f7590ee77509c81ac315c2
>
> Useful for context, but the code can probably be tossed at this point.

OK!

>> That seems like a good starting point.  But I know David feels that with
>> real-world experience in devops type work that actually things would be
>> a bit different than what's described in his original email.  I'm not
>> sure myself what would be different.  It would be helpful to hear Dave
>> weigh in on that point.
>
> Sure, since 2015 I've become the lead devops person at my company, so
> I like to think I'm a bit wiser now.

:)

>> Maybe Dave and I can meet up IRL now that we're close enough to each
>> other to chat about it.  But I know it's less fun than it used to be for
>> Dave to consider this because now that's Dave's actual job... but all
>> the more reason we need Dave's wisdom! :)
>
> We could meet up IRL about this and I can try to make an earnest
> effort to deal with this. I think what has stopped me in the past is
> the sheer size of this project, and maybe dramatically scaling down
> the scope will allow us to get *something* out the door.

Yes I think we shouldn't allow ourselves to get too overwhelmed!  We
should give the warnings that this is an alpha system in alpha (soon
beta?) software and we don't guarantee that the api will be stable :)

Furthermore, I think if we put an emphasis on composable layers, and
start with the simplest lower layers, and then add some other layers
above it (and actually the different use cases you lay out may actually
swap out which layers are above this), then I think we could be good?

> Here are some general use-cases I know about for deployments, roughly
> ordered from small scale to large scale, and least complex to most
> complex:
>
> * Managing a physical machine or two that have been given memorable
> names that you update in-place (home scale)
> * Managing a virtual machine or two that have been given memorable
> names that you update in-place (blog scale)

In the past we have discussed storing "deployment state", and where and
how we would store it.  I will make the argument that for these two
cases, we actually don't need state!  We can just do the simplest thing
possible: build the system derivation for each of these machines, log in
and see if the system already has that derivation running, and if not
copy over the closure and install it as the system state.  This could
be entirely declarative.

I think initially we can ignore entirely the issue of provisioning an
initial guix image.  We can also ignore things like getting the right
public keys and configuring the initial authority of how you can be
authorized to copy over that data.

> * Managing a large number of virtual machines whose names don't matter
> that you update in-place (proto-cloud scale)
> * Managing a large number of virtual machines whose names don't matter
> that are replaced when there is an update (cloud scale)
> * Managing 1 or more clusters of physical machines (datacenter scale)
> * Managing 1 or more clusters of physical machines and virtual
> machines ("corporation with a datacenter that is moving some stuff to
> the cloud" scale)

For all these, I think there can be one or more abstraction layers that
build on top of the same declarative primitives described above.  And
this may be where state comes in; since you don't really know what the
identifiers are ahead of time, but you do need to keep track of them,
yes we do want to serialize the state somewhere.

> There are, of course, more scenarios to consider (haven't even touched
> upon things like a Kubernetes cluster), but this is enough to
> illustrate the point that is a great diversity in setups.  How many
> machines are there? Are the bare metal, virtual machines, or a mix of
> both? In the case of virtual machines, are updates applied in an
> immutable fashion or not?  If immutable, which technique (blue-green,
> rolling release, etc.)?  It makes my head spin to think about all the
> use-cases.

... I don't know anything about the things you said above so I guess
that would be nice to go over in person :)

I think maybe my ignorance is helpful for optimism in this case!

> So... let's start small. Can we write a tool that handles in-place
> updates to machines (physical or virtual) whose name and IP address we
> know well (our special pet servers) without precluding the possibility
> of scaling up to more sophisticated architectures?  This would address
> the "home" and "blog" scale items above, which is probably what most
> of the people actually using Guix today would want.  I got stuck
> trying to do in-place updates to remote machines years ago, but that
> was before Ludo made it easy to connect to remote systems.

Yes!

Also I think guile-ssh might make things a bit easier than they were in
the past, too.

Let's do this!

Re: It's time to build "guix deploy"

[Thompson, David]

On Tue, Feb 12, 2019 at 8:34 AM Christopher Lemmer Webber
<cwebber@dustycloud.org> wrote:
>
> Thompson, David writes:
>
> >> Maybe Dave and I can meet up IRL now that we're close enough to each
> >> other to chat about it.  But I know it's less fun than it used to be for
> >> Dave to consider this because now that's Dave's actual job... but all
> >> the more reason we need Dave's wisdom! :)
> >
> > We could meet up IRL about this and I can try to make an earnest
> > effort to deal with this. I think what has stopped me in the past is
> > the sheer size of this project, and maybe dramatically scaling down
> > the scope will allow us to get *something* out the door.
>
> Yes I think we shouldn't allow ourselves to get too overwhelmed!  We
> should give the warnings that this is an alpha system in alpha (soon
> beta?) software and we don't guarantee that the api will be stable :)
>
> Furthermore, I think if we put an emphasis on composable layers, and
> start with the simplest lower layers, and then add some other layers
> above it (and actually the different use cases you lay out may actually
> swap out which layers are above this), then I think we could be good?

Something like that, yeah.

> > Here are some general use-cases I know about for deployments, roughly
> > ordered from small scale to large scale, and least complex to most
> > complex:
> >
> > * Managing a physical machine or two that have been given memorable
> > names that you update in-place (home scale)
> > * Managing a virtual machine or two that have been given memorable
> > names that you update in-place (blog scale)
>
> In the past we have discussed storing "deployment state", and where and
> how we would store it.  I will make the argument that for these two
> cases, we actually don't need state!  We can just do the simplest thing
> possible: build the system derivation for each of these machines, log in
> and see if the system already has that derivation running, and if not
> copy over the closure and install it as the system state.  This could
> be entirely declarative.

Right, since these are pet servers, the user already knows all the
relevant information about them.

> I think initially we can ignore entirely the issue of provisioning an
> initial guix image.  We can also ignore things like getting the right
> public keys and configuring the initial authority of how you can be
> authorized to copy over that data.

Yes, the pet server use-cases assume the systems have been setup
manually.  Automated provisioning will be relevant for cloud
deployments.

> > * Managing a large number of virtual machines whose names don't matter
> > that you update in-place (proto-cloud scale)
> > * Managing a large number of virtual machines whose names don't matter
> > that are replaced when there is an update (cloud scale)
> > * Managing 1 or more clusters of physical machines (datacenter scale)
> > * Managing 1 or more clusters of physical machines and virtual
> > machines ("corporation with a datacenter that is moving some stuff to
> > the cloud" scale)
>
> For all these, I think there can be one or more abstraction layers that
> build on top of the same declarative primitives described above.  And
> this may be where state comes in; since you don't really know what the
> identifiers are ahead of time, but you do need to keep track of them,
> yes we do want to serialize the state somewhere.

I don't think this tool should ever store state.  For example, if you
have a bunch of EC2 instances, you can use the AWS API to query for
the instances you want to update.  No need for us to worry about state
and deal with the inevitable synchronization issues.

> > There are, of course, more scenarios to consider (haven't even touched
> > upon things like a Kubernetes cluster), but this is enough to
> > illustrate the point that is a great diversity in setups.  How many
> > machines are there? Are the bare metal, virtual machines, or a mix of
> > both? In the case of virtual machines, are updates applied in an
> > immutable fashion or not?  If immutable, which technique (blue-green,
> > rolling release, etc.)?  It makes my head spin to think about all the
> > use-cases.
>
> ... I don't know anything about the things you said above so I guess
> that would be nice to go over in person :)
>
> I think maybe my ignorance is helpful for optimism in this case!
>
> > So... let's start small. Can we write a tool that handles in-place
> > updates to machines (physical or virtual) whose name and IP address we
> > know well (our special pet servers) without precluding the possibility
> > of scaling up to more sophisticated architectures?  This would address
> > the "home" and "blog" scale items above, which is probably what most
> > of the people actually using Guix today would want.  I got stuck
> > trying to do in-place updates to remote machines years ago, but that
> > was before Ludo made it easy to connect to remote systems.
>
> Yes!
>
> Also I think guile-ssh might make things a bit easier than they were in
> the past, too.
>
> Let's do this!

Sounds good!

- Dave

building "guix deploy"

[Thompson, David]

Hello everyone!

Chris Webber and I spent the morning chatting about how we want to
approach making "guix deploy" a reality and then started hacking on it
in the afternoon.  Although we weren't able to complete a working
prototype by the end of the day, we were able to get pretty close.  We
created a 'guix deploy' CLI to build derivations for any number of
remote systems on a local workstation and initiate the transfer to the
remote systems, but we encountered a difficult to debug SSH error that
blocked our progress:

sending 85 store items (0 MiB) to 'test.activitypub.rocks'...
exporting path `/gnu/store/ylcdmrj3vf00ixdcjvkl3mbs8f5i9w8l-git-minimal-2.20.1.drv'
;;; [2019/03/09 17:32:48.792589, 0] write_to_channel_port: [GSSH
ERROR] Remote channel is closed: #<input-output: channel (open)
541a220>
Backtrace:
          10 (apply-smob/1 #<catch-closure a26900>)
In ice-9/boot-9.scm:
    705:2  9 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
In ice-9/eval.scm:
    619:8  8 (_ #(#(#<directory (guile-user) ab1140>)))
In guix/ui.scm:
  1654:12  7 (run-guix-command _ . _)
In guix/scripts/deploy.scm:
     72:8  6 (guix-deploy . _)
In srfi/srfi-1.scm:
    640:9  5 (for-each #<procedure 4e5c940 at guix/scripts/deploy.s…> …)
In guix/scripts/deploy.scm:
    74:20  4 (_ _)
In gnu/machine.scm:
    58:22  3 (_ _ _ _)
In guix/ssh.scm:
    313:4  2 (send-files #<store-connection 256.99 48000f0> _ _ # _ # …)
In guix/store.scm:
   1504:7  1 (export-paths #<store-connection 256.99 48000f0> _ #<i…> …)
In unknown file:
           0 (put-bytevector #<input-output: channel (open) 541a220> …)

ERROR: In procedure put-bytevector:
Throw to key `guile-ssh-error' with args `("write_to_channel_port"
"Remote channel is closed" #<input-output: channel (open) 541a220>
#f)'.

If anyone knows what might be going on here and how we could resolve
it, your input would be much appreciated!  We verified via the sshd
logs that we were indeed successfully establishing a connection.

Once we're past this blocking issue and are able to transfer OS
closures to remote systems, we plan to write a modified version of
switch-to-system that uses guile-ssh to switch remote symlinks for the
active system and run the activation script.  We'll save
upgrade-shepherd-services for later, as it is quite a bit more
complex.

There's not a lot of code yet, but you can check it out in the
wip-deploy2 branch.  Currently, the only supported use-case is running
the equivalent of 'guix system reconfigure' on machines already
running GuixSD that have an OpenSSH daemon running, but the basic
design allows for additional use-cases to be supported in the future.

Thanks to Ricardo for his install-berlin.scm script that provided a
basis for most of our code, and to Ludovic for the (guix ssh) module
that I wish I had when I attempted to write 'guix deploy' all those
years ago.

Until next time...

- Dave

Re: building "guix deploy"

[Ludovic Courtès]

Hi there!

"Thompson, David" <dthompson2@worcester.edu> skribis:

> Chris Webber and I spent the morning chatting about how we want to
> approach making "guix deploy" a reality and then started hacking on it
> in the afternoon.  Although we weren't able to complete a working
> prototype by the end of the day, we were able to get pretty close.  We
> created a 'guix deploy' CLI to build derivations for any number of
> remote systems on a local workstation and initiate the transfer to the
> remote systems, but we encountered a difficult to debug SSH error that
> blocked our progress:
>
> sending 85 store items (0 MiB) to 'test.activitypub.rocks'...
> exporting path `/gnu/store/ylcdmrj3vf00ixdcjvkl3mbs8f5i9w8l-git-minimal-2.20.1.drv'
> ;;; [2019/03/09 17:32:48.792589, 0] write_to_channel_port: [GSSH
> ERROR] Remote channel is closed: #<input-output: channel (open)
> 541a220>
> Backtrace:
>           10 (apply-smob/1 #<catch-closure a26900>)
> In ice-9/boot-9.scm:
>     705:2  9 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
> In ice-9/eval.scm:
>     619:8  8 (_ #(#(#<directory (guile-user) ab1140>)))
> In guix/ui.scm:
>   1654:12  7 (run-guix-command _ . _)
> In guix/scripts/deploy.scm:
>      72:8  6 (guix-deploy . _)
> In srfi/srfi-1.scm:
>     640:9  5 (for-each #<procedure 4e5c940 at guix/scripts/deploy.s…> …)
> In guix/scripts/deploy.scm:
>     74:20  4 (_ _)
> In gnu/machine.scm:
>     58:22  3 (_ _ _ _)
> In guix/ssh.scm:
>     313:4  2 (send-files #<store-connection 256.99 48000f0> _ _ # _ # …)
> In guix/store.scm:
>    1504:7  1 (export-paths #<store-connection 256.99 48000f0> _ #<i…> …)
> In unknown file:
>            0 (put-bytevector #<input-output: channel (open) 541a220> …)
>
> ERROR: In procedure put-bytevector:
> Throw to key `guile-ssh-error' with args `("write_to_channel_port"
> "Remote channel is closed" #<input-output: channel (open) 541a220>
> #f)'.
>
> If anyone knows what might be going on here and how we could resolve
> it, your input would be much appreciated!  We verified via the sshd
> logs that we were indeed successfully establishing a connection.

Error reporting in (guix ssh) is, ahem, not as good as it could be.

Apparently the SSH channel was closed prematurely, which could be due to
a number of things:

  1. Are ‘guix’ and ‘guile’ in $PATH on the remote machine, for
     non-interactive shells?

  2. Is ‘guix repl’ available in the remote machine?

You can test this with:

  ssh HOST guile --version
  ssh HOST guix repl --version

Also, does ‘guix copy’ fail similarly when sending files to that host?

> Once we're past this blocking issue and are able to transfer OS
> closures to remote systems, we plan to write a modified version of
> switch-to-system that uses guile-ssh to switch remote symlinks for the
> active system and run the activation script.  We'll save
> upgrade-shepherd-services for later, as it is quite a bit more
> complex.

My plan is to have ‘guix system reconfigure --host=host.example.org’.
To do that, I thought about the following plan:

  1. Isolate the effectful part of reconfigure (basically
     ‘switch-to-system’).

  2. Implement ‘remote-eval’, which takes a gexp and an SSH session and
     evaluates the expression remotely, copying the gexp inputs as
     needed.

  3. Have ‘reconfigure’ use either ‘eval’ or ‘remote-eval’ to evaluate
     the effectful bits of reconfigure.

#1 is a bit annoying because we need to untangle code so that we can
easily put it all “on the build side.”  In particular, I think we’ll
have to change (guix graph), used by ‘upgrade-shepherd-services’, so
that it no longer depends on ‘%store-monad’.

That said, it’s probably a good idea to take a shorter path in the
meantime to unlock progress on ‘guix deploy’!

> There's not a lot of code yet, but you can check it out in the
> wip-deploy2 branch.  Currently, the only supported use-case is running
> the equivalent of 'guix system reconfigure' on machines already
> running GuixSD that have an OpenSSH daemon running, but the basic
> design allows for additional use-cases to be supported in the future.

Yay!

Thank you gentlefolks for resuming work on this!

Ludo’.

Re: building "guix deploy"

[Christopher Lemmer Webber]

Ludovic Courtès writes:

> Hi there!

Hi Ludo!  Thanks for your reply.

> "Thompson, David" <dthompson2@worcester.edu> skribis:
>
>> Chris Webber and I spent the morning chatting about how we want to
>> approach making "guix deploy" a reality and then started hacking on it
>> in the afternoon.  Although we weren't able to complete a working
>> prototype by the end of the day, we were able to get pretty close.  We
>> created a 'guix deploy' CLI to build derivations for any number of
>> remote systems on a local workstation and initiate the transfer to the
>> remote systems, but we encountered a difficult to debug SSH error that
>> blocked our progress:
>>
>> sending 85 store items (0 MiB) to 'test.activitypub.rocks'...
>> exporting path `/gnu/store/ylcdmrj3vf00ixdcjvkl3mbs8f5i9w8l-git-minimal-2.20.1.drv'
>> ;;; [2019/03/09 17:32:48.792589, 0] write_to_channel_port: [GSSH
>> ERROR] Remote channel is closed: #<input-output: channel (open)
>> 541a220>
>> Backtrace:
>>           10 (apply-smob/1 #<catch-closure a26900>)
>> In ice-9/boot-9.scm:
>>     705:2  9 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
>> In ice-9/eval.scm:
>>     619:8  8 (_ #(#(#<directory (guile-user) ab1140>)))
>> In guix/ui.scm:
>>   1654:12  7 (run-guix-command _ . _)
>> In guix/scripts/deploy.scm:
>>      72:8  6 (guix-deploy . _)
>> In srfi/srfi-1.scm:
>>     640:9  5 (for-each #<procedure 4e5c940 at guix/scripts/deploy.s…> …)
>> In guix/scripts/deploy.scm:
>>     74:20  4 (_ _)
>> In gnu/machine.scm:
>>     58:22  3 (_ _ _ _)
>> In guix/ssh.scm:
>>     313:4  2 (send-files #<store-connection 256.99 48000f0> _ _ # _ # …)
>> In guix/store.scm:
>>    1504:7  1 (export-paths #<store-connection 256.99 48000f0> _ #<i…> …)
>> In unknown file:
>>            0 (put-bytevector #<input-output: channel (open) 541a220> …)
>>
>> ERROR: In procedure put-bytevector:
>> Throw to key `guile-ssh-error' with args `("write_to_channel_port"
>> "Remote channel is closed" #<input-output: channel (open) 541a220>
>> #f)'.
>>
>> If anyone knows what might be going on here and how we could resolve
>> it, your input would be much appreciated!  We verified via the sshd
>> logs that we were indeed successfully establishing a connection.
>
> Error reporting in (guix ssh) is, ahem, not as good as it could be.
>
> Apparently the SSH channel was closed prematurely, which could be due to
> a number of things:
>
>   1. Are ‘guix’ and ‘guile’ in $PATH on the remote machine, for
>      non-interactive shells?
>
>   2. Is ‘guix repl’ available in the remote machine?
>
> You can test this with:
>
>   ssh HOST guile --version
>   ssh HOST guix repl --version

Yep, both respond with
  guile (GNU Guile) 2.2.4
and
  guix (GNU Guix) 0.16.0-10.2637cfd
respectively.

> Also, does ‘guix copy’ fail similarly when sending files to that host?

It seems it does:

cwebber@jasmine:~/devel/librelounge-audio$ guix copy --to=test.activitypub.rocks pidgin
guile: warning: failed to install locale
sending 37 store items (336 MiB) to 'test.activitypub.rocks'...
;;; [2019/03/11 10:39:25.573104, 0] write_to_channel_port: [GSSH ERROR] Remote channel is closed: #<input-output: channel (open) 46f5e60>
Backtrace:
          11 (primitive-load "/home/cwebber/.config/guix/current/bin…")
In guix/ui.scm:
  1654:12 10 (run-guix-command _ . _)
In ice-9/boot-9.scm:
    829:9  9 (catch _ _ #<procedure 7fca74f959b8 at guix/ui.scm:624…> …)
    829:9  8 (catch _ _ #<procedure 7fca74f959d0 at guix/ui.scm:750…> …)
In guix/status.scm:
    810:4  7 (call-with-status-report _ _)
In guix/scripts/copy.scm:
    81:27  6 (send-to-remote-host _ _)
In guix/ssh.scm:
    313:4  5 (send-files #<store-connection 256.97 27a5eb0> _ _ # _ # …)
In guix/store.scm:
  1505:12  4 (export-paths #<store-connection 256.97 27a5eb0> _ #<i…> …)
  1485:22  3 (export-path #<store-connection 256.97 27a5eb0> _ #<in…> …)
   683:13  2 (process-stderr _ _)
   646:10  1 (dump-port #<input-output: socket 14> #<input-output: …> …)
In unknown file:
           0 (put-bytevector #<input-output: channel (open) 46f5e60> …)

ERROR: In procedure put-bytevector:
Throw to key `guile-ssh-error' with args `("write_to_channel_port" "Remote channel is closed" #<input-output: channel (open) 46f5e60> #f)'.

I wonder what got screwed up!

>> Once we're past this blocking issue and are able to transfer OS
>> closures to remote systems, we plan to write a modified version of
>> switch-to-system that uses guile-ssh to switch remote symlinks for the
>> active system and run the activation script.  We'll save
>> upgrade-shepherd-services for later, as it is quite a bit more
>> complex.
>
> My plan is to have ‘guix system reconfigure --host=host.example.org’.
> To do that, I thought about the following plan:
>
>   1. Isolate the effectful part of reconfigure (basically
>      ‘switch-to-system’).
>
>   2. Implement ‘remote-eval’, which takes a gexp and an SSH session and
>      evaluates the expression remotely, copying the gexp inputs as
>      needed.
>
>   3. Have ‘reconfigure’ use either ‘eval’ or ‘remote-eval’ to evaluate
>      the effectful bits of reconfigure.

This sounds like the right approach to me.

> #1 is a bit annoying because we need to untangle code so that we can
> easily put it all “on the build side.”  In particular, I think we’ll
> have to change (guix graph), used by ‘upgrade-shepherd-services’, so
> that it no longer depends on ‘%store-monad’.
>
> That said, it’s probably a good idea to take a shorter path in the
> meantime to unlock progress on ‘guix deploy’!
>
>> There's not a lot of code yet, but you can check it out in the
>> wip-deploy2 branch.  Currently, the only supported use-case is running
>> the equivalent of 'guix system reconfigure' on machines already
>> running GuixSD that have an OpenSSH daemon running, but the basic
>> design allows for additional use-cases to be supported in the future.
>
> Yay!
>
> Thank you gentlefolks for resuming work on this!
>
> Ludo’.

Thank you for your response!

By the way, since the problem somehow seems to be a problem with my
server and ssh + guile-ssh (!!!) I wonder if someone else has a server
set up they'd be brave enough to try the above branch with?  It appears
that the problem is not Dave's code, given that "guix copy" also fails.

Re: building "guix deploy"

[Ludovic Courtès]

Howdy!

Christopher Lemmer Webber <cwebber@dustycloud.org> skribis:

> Ludovic Courtès writes:

[...]

>> Error reporting in (guix ssh) is, ahem, not as good as it could be.
>>
>> Apparently the SSH channel was closed prematurely, which could be due to
>> a number of things:
>>
>>   1. Are ‘guix’ and ‘guile’ in $PATH on the remote machine, for
>>      non-interactive shells?
>>
>>   2. Is ‘guix repl’ available in the remote machine?
>>
>> You can test this with:
>>
>>   ssh HOST guile --version
>>   ssh HOST guix repl --version
>
> Yep, both respond with
>   guile (GNU Guile) 2.2.4
> and
>   guix (GNU Guix) 0.16.0-10.2637cfd
> respectively.
>
>> Also, does ‘guix copy’ fail similarly when sending files to that host?
>
> It seems it does:
>
> cwebber@jasmine:~/devel/librelounge-audio$ guix copy --to=test.activitypub.rocks pidgin
> guile: warning: failed to install locale
> sending 37 store items (336 MiB) to 'test.activitypub.rocks'...
> ;;; [2019/03/11 10:39:25.573104, 0] write_to_channel_port: [GSSH ERROR] Remote channel is closed: #<input-output: channel (open) 46f5e60>
> Backtrace:
>           11 (primitive-load "/home/cwebber/.config/guix/current/bin…")
> In guix/ui.scm:
>   1654:12 10 (run-guix-command _ . _)
> In ice-9/boot-9.scm:
>     829:9  9 (catch _ _ #<procedure 7fca74f959b8 at guix/ui.scm:624…> …)
>     829:9  8 (catch _ _ #<procedure 7fca74f959d0 at guix/ui.scm:750…> …)
> In guix/status.scm:
>     810:4  7 (call-with-status-report _ _)
> In guix/scripts/copy.scm:
>     81:27  6 (send-to-remote-host _ _)
> In guix/ssh.scm:
>     313:4  5 (send-files #<store-connection 256.97 27a5eb0> _ _ # _ # …)
> In guix/store.scm:
>   1505:12  4 (export-paths #<store-connection 256.97 27a5eb0> _ #<i…> …)
>   1485:22  3 (export-path #<store-connection 256.97 27a5eb0> _ #<in…> …)
>    683:13  2 (process-stderr _ _)
>    646:10  1 (dump-port #<input-output: socket 14> #<input-output: …> …)
> In unknown file:
>            0 (put-bytevector #<input-output: channel (open) 46f5e60> …)
>
> ERROR: In procedure put-bytevector:
> Throw to key `guile-ssh-error' with args `("write_to_channel_port" "Remote channel is closed" #<input-output: channel (open) 46f5e60> #f)'.
>
> I wonder what got screwed up!

Could you, on test.activitypub.rocks, do something along these lines:

  sudo strace -p PID -s 300 -o log -f

where PID is the PID of the main ‘sshd’ process.

And after that, re-run ‘guix copy’, and grab the ‘log’.

Thanks,
Ludo’.