Progress on 'guix deploy'

Progress on 'guix deploy'

[Jakob L. Kreuze]

[-- Attachment #1: Type: text/plain, Size: 6023 bytes --]

Hello, Guix!

Apart from a few patches and my introductory email about a month ago,
I've been pretty silent. I feel it's time to finally break that silence,
let people know where progress has been made, and request some feedback
on the code I've written so far.

As a quick refresher, my GSoC project this summer is 'guix deploy', a
deployment automation tool for GuixSD that's been discussed more
thoroughly in [1] and [2]. Development has taken place on my personal
branch of Guix, specifically the 'wip-deploy' branch [3], and is
represented by three new Scheme source files:

- 'gnu/machine.scm', which provides an abstraction for /something/ that
  can be deployed to in a heterogeneous deployment. Currently, the only
  concrete implementation of this is the simple case of in-place updates
  to machines running SSH whose names and IP addresses we know.
- 'gnu/tests/machine.scm', which implements some tests for
  'gnu/machine.scm'. This is where I'm most interested in receiving
  feedback. More on that later.
- 'guix/scripts/deploy.scm', which implements the rudimentary
  command-line interface for 'guix deploy'.

The command-line interface hasn't really been fleshed out yet, but if
you'd like to play around with it, it takes a "deployment" file as a
parameter, which is a Scheme file looking something like the following:

#+BEGIN_SRC scheme
(use-modules ...)

(define %system
  (operating-system
   (host-name "gnu-deployed")
   (timezone "Etc/UTC")
   (bootloader (bootloader-configuration
                (bootloader grub-bootloader)
                (target "/dev/sda")
                (terminal-outputs '(console))))
   (file-systems (cons (file-system
                        (mount-point "/")
                        (device "/dev/vda1")
                        (type "ext4"))
                       %base-file-systems))
   (services
    (append (list (service dhcp-client-service-type)
                  (service openssh-service-type
                           (openssh-configuration
                            (permit-root-login #t)
                            (allow-empty-passwords? #t))))
            %base-services))))

(list (make <sshable-machine>
        #:host-name "localhost"
        #:ssh-port 5556
        #:system %system))
#+END_SRC

Basically, you attach an 'operating-system' declaration to a 'machine'.
In this case, 'sshable-machine' is the specific type of 'machine' that
we're deploying to (one that's running an SSH daemon and has a known IP
+ port + hostname).

I've found that the GuixSD QEMU images work well for playing around with
this, provided that you add the SSH service to the system configuration
and start it. In the case of this deployment file, I had a GuixSD guest
with port 22 forwarded to my host's port 5556. You'll also need to set
up some sort of public key auth in your SSH config. The current code
isn't capable of handling other forms of SSH authentication.

In terms of implementation, GOOPS feels like a bit of an unusual choice
when juxtaposed with the rest of the Guix codebase, but I've come to
really enjoy it. I'll roll with it for now, since I think it will make
it easier to flesh out the vocabulary for specifying deployments.

The implementation of '<sshable-machine>' is doing what
'switch-to-system' and 'install-bootloader' in 'guix/scripts/system.scm'
do, but in terms of data that can be sent with 'remote-eval'. I imagine
the code will make more sense if you read both simultaneously.

Okay, on to the test suite.

My understanding of the system test suite (tests run with 'check-system'
as opposed to those run with 'check') is that the meat of the test code
exists in a G-Expression and should _not_ be interacting with the store
of the machine running the test suite (i.e. that's the reason we're
using marionettes in the first place). 'gnu/tests/install.scm' seems to
be somewhat of an exception, and because the code in '(gnu machine)'
depends heavily on having access to a store, I've tried to extend what's
done in 'gnu/tests/install.scm' so that my tests have access to store
while instrumenting the marionettes.

To be specific, the chicken and egg scenario I'm working around is that
the SSH daemon on the marionette needs to be running in order for
'deploy-os' to work, but I can't call 'deploy-os' from the test
G-Expression because the store wouldn't be accessible then.

My gut is telling me that this is absolutely the wrong way to go about
it, though. 'call-with-marionette' is one of a couple red flags making
me feel that way -- I don't think marionettes were meant to be started
outside the context of a derivation...

If anyone has suggestions on how I might go about properly testing this
code, I would appreciate it very much.

Another point about the test suite: the 'ssh-deploy-os' test fails, but
it's a reproducible version of the issue outlined in [2]. I'd like to
conscript some help from those more familiar with guile-ssh before
breaking out the ol' RFCs myself.

...

So, if anyone's in the mood to peek at what's been written so far and
give feedback, that'd make my day. Doesn't have to be feedback related
to the test code, either. Any sort of comment, be it regarding the code
style or perhaps even some suggestions on improving the interface, would
be appreciated.

I'm going to continue to spend time on the "internals" of 'guix deploy'
this coming week, incorporating any feedback I receive and ensuring that
I have a framework to build upon when I extend this to more complicated
scenarios like deploying to a VPS provider. After that, I'll tackle
fleshing out the way deployments are specified (the file part. I'm
holding off on the command-line tool part for right now).

Signing off for now. Huge thanks to everyone for the warm welcome I
received following my introduction email.

Regards,
Jakob

[1]: https://lists.gnu.org/archive/html/guix-devel/2019-02/msg00145.html
[2]: https://lists.gnu.org/archive/html/guix-devel/2019-03/msg00114.html
[3]: https://git.sr.ht/~jakob/guix/tree/wip-deploy

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Progress on 'guix deploy'

[Christopher Lemmer Webber]

Jakob L. Kreuze writes:

> Hello, Guix!
>
> Apart from a few patches and my introductory email about a month ago,
> I've been pretty silent. I feel it's time to finally break that silence,
> let people know where progress has been made, and request some feedback
> on the code I've written so far.

Hi Jakob!  Thanks for sending this to the list. :)

> As a quick refresher, my GSoC project this summer is 'guix deploy', a
> deployment automation tool for GuixSD that's been discussed more
> thoroughly in [1] and [2]. Development has taken place on my personal
> branch of Guix, specifically the 'wip-deploy' branch [3], and is
> represented by three new Scheme source files:
>
> - 'gnu/machine.scm', which provides an abstraction for /something/ that
>   can be deployed to in a heterogeneous deployment. Currently, the only
>   concrete implementation of this is the simple case of in-place updates
>   to machines running SSH whose names and IP addresses we know.
> - 'gnu/tests/machine.scm', which implements some tests for
>   'gnu/machine.scm'. This is where I'm most interested in receiving
>   feedback. More on that later.
> - 'guix/scripts/deploy.scm', which implements the rudimentary
>   command-line interface for 'guix deploy'.
>
> The command-line interface hasn't really been fleshed out yet, but if
> you'd like to play around with it, it takes a "deployment" file as a
> parameter, which is a Scheme file looking something like the following:
>
> #+BEGIN_SRC scheme
> (use-modules ...)
>
> (define %system
>   (operating-system
>    (host-name "gnu-deployed")
>    (timezone "Etc/UTC")
>    (bootloader (bootloader-configuration
>                 (bootloader grub-bootloader)
>                 (target "/dev/sda")
>                 (terminal-outputs '(console))))
>    (file-systems (cons (file-system
>                         (mount-point "/")
>                         (device "/dev/vda1")
>                         (type "ext4"))
>                        %base-file-systems))
>    (services
>     (append (list (service dhcp-client-service-type)
>                   (service openssh-service-type
>                            (openssh-configuration
>                             (permit-root-login #t)
>                             (allow-empty-passwords? #t))))
>             %base-services))))
>
> (list (make <sshable-machine>
>         #:host-name "localhost"
>         #:ssh-port 5556
>         #:system %system))
> #+END_SRC
>
> Basically, you attach an 'operating-system' declaration to a 'machine'.
> In this case, 'sshable-machine' is the specific type of 'machine' that
> we're deploying to (one that's running an SSH daemon and has a known IP
> + port + hostname).
>
> I've found that the GuixSD QEMU images work well for playing around with
> this, provided that you add the SSH service to the system configuration
> and start it. In the case of this deployment file, I had a GuixSD guest
> with port 22 forwarded to my host's port 5556. You'll also need to set
> up some sort of public key auth in your SSH config. The current code
> isn't capable of handling other forms of SSH authentication.
>
> In terms of implementation, GOOPS feels like a bit of an unusual choice
> when juxtaposed with the rest of the Guix codebase, but I've come to
> really enjoy it. I'll roll with it for now, since I think it will make
> it easier to flesh out the vocabulary for specifying deployments.

I also want to clarify: I know that GOOPS code isn't quite contemporary
style for Guix, but David Thompson and I encouraged Jakob to spend the
minimal amount of time necessary focusing on doing polymorphism in a
clever way, as this is what Dave and I found was slowing us down in our
initial prototype code.

It's possible we could/should move to something else, but that's the
motivation there.

> The implementation of '<sshable-machine>' is doing what
> 'switch-to-system' and 'install-bootloader' in 'guix/scripts/system.scm'
> do, but in terms of data that can be sent with 'remote-eval'. I imagine
> the code will make more sense if you read both simultaneously.
>
> Okay, on to the test suite.
>
> My understanding of the system test suite (tests run with 'check-system'
> as opposed to those run with 'check') is that the meat of the test code
> exists in a G-Expression and should _not_ be interacting with the store
> of the machine running the test suite (i.e. that's the reason we're
> using marionettes in the first place). 'gnu/tests/install.scm' seems to
> be somewhat of an exception, and because the code in '(gnu machine)'
> depends heavily on having access to a store, I've tried to extend what's
> done in 'gnu/tests/install.scm' so that my tests have access to store
> while instrumenting the marionettes.
>
> To be specific, the chicken and egg scenario I'm working around is that
> the SSH daemon on the marionette needs to be running in order for
> 'deploy-os' to work, but I can't call 'deploy-os' from the test
> G-Expression because the store wouldn't be accessible then.
>
> My gut is telling me that this is absolutely the wrong way to go about
> it, though. 'call-with-marionette' is one of a couple red flags making
> me feel that way -- I don't think marionettes were meant to be started
> outside the context of a derivation...
>
> If anyone has suggestions on how I might go about properly testing this
> code, I would appreciate it very much.
>
> Another point about the test suite: the 'ssh-deploy-os' test fails, but
> it's a reproducible version of the issue outlined in [2]. I'd like to
> conscript some help from those more familiar with guile-ssh before
> breaking out the ol' RFCs myself.
>
> ...
>
> So, if anyone's in the mood to peek at what's been written so far and
> give feedback, that'd make my day. Doesn't have to be feedback related
> to the test code, either. Any sort of comment, be it regarding the code
> style or perhaps even some suggestions on improving the interface, would
> be appreciated.
>
> I'm going to continue to spend time on the "internals" of 'guix deploy'
> this coming week, incorporating any feedback I receive and ensuring that
> I have a framework to build upon when I extend this to more complicated
> scenarios like deploying to a VPS provider. After that, I'll tackle
> fleshing out the way deployments are specified (the file part. I'm
> holding off on the command-line tool part for right now).
>
> Signing off for now. Huge thanks to everyone for the warm welcome I
> received following my introduction email.
>
> Regards,
> Jakob
>
> [1]: https://lists.gnu.org/archive/html/guix-devel/2019-02/msg00145.html
> [2]: https://lists.gnu.org/archive/html/guix-devel/2019-03/msg00114.html
> [3]: https://git.sr.ht/~jakob/guix/tree/wip-deploy

FWIW I could use the help as well; Jakob is hitting some areas of code
I'm not very familiar with and unfortunately I've been traveling for the
last two weeks for some family stuff.  I'm keeping in good contact with
Jakob and talking with him about problems but more community code review
will be helpful until at least next week when I'll be able to hit the
ground running again.

Jakob has been very communicative btw, and I'm really optimistic about
the state of the project!

 - Chris

Re: Progress on 'guix deploy'

[Ludovic Courtès]

Hello Jakob,

zerodaysfordays@sdf.lonestar.org (Jakob L. Kreuze) skribis:

> As a quick refresher, my GSoC project this summer is 'guix deploy', a
> deployment automation tool for GuixSD that's been discussed more
> thoroughly in [1] and [2]. Development has taken place on my personal
> branch of Guix, specifically the 'wip-deploy' branch [3], and is
> represented by three new Scheme source files:
>
> - 'gnu/machine.scm', which provides an abstraction for /something/ that
>   can be deployed to in a heterogeneous deployment. Currently, the only
>   concrete implementation of this is the simple case of in-place updates
>   to machines running SSH whose names and IP addresses we know.
> - 'gnu/tests/machine.scm', which implements some tests for
>   'gnu/machine.scm'. This is where I'm most interested in receiving
>   feedback. More on that later.
> - 'guix/scripts/deploy.scm', which implements the rudimentary
>   command-line interface for 'guix deploy'.

Nice!

> The command-line interface hasn't really been fleshed out yet, but if
> you'd like to play around with it, it takes a "deployment" file as a
> parameter, which is a Scheme file looking something like the following:
>
> #+BEGIN_SRC scheme
> (use-modules ...)
>
> (define %system
>   (operating-system
>    (host-name "gnu-deployed")
>    (timezone "Etc/UTC")
>    (bootloader (bootloader-configuration
>                 (bootloader grub-bootloader)
>                 (target "/dev/sda")
>                 (terminal-outputs '(console))))
>    (file-systems (cons (file-system
>                         (mount-point "/")
>                         (device "/dev/vda1")
>                         (type "ext4"))
>                        %base-file-systems))
>    (services
>     (append (list (service dhcp-client-service-type)
>                   (service openssh-service-type
>                            (openssh-configuration
>                             (permit-root-login #t)
>                             (allow-empty-passwords? #t))))
>             %base-services))))
>
> (list (make <sshable-machine>
>         #:host-name "localhost"
>         #:ssh-port 5556
>         #:system %system))
> #+END_SRC
>
> Basically, you attach an 'operating-system' declaration to a 'machine'.
> In this case, 'sshable-machine' is the specific type of 'machine' that
> we're deploying to (one that's running an SSH daemon and has a known IP
> + port + hostname).

Looks great to me.

> I've found that the GuixSD QEMU images work well for playing around with
> this, provided that you add the SSH service to the system configuration
> and start it. In the case of this deployment file, I had a GuixSD guest
> with port 22 forwarded to my host's port 5556. You'll also need to set
> up some sort of public key auth in your SSH config. The current code
> isn't capable of handling other forms of SSH authentication.

That’s reasonable.

> In terms of implementation, GOOPS feels like a bit of an unusual choice
> when juxtaposed with the rest of the Guix codebase, but I've come to
> really enjoy it. I'll roll with it for now, since I think it will make
> it easier to flesh out the vocabulary for specifying deployments.

It’s more than unusual.  :-)  Perhaps it’s the right choice here, we’ll
have to discuss it at some point!

> The implementation of '<sshable-machine>' is doing what
> 'switch-to-system' and 'install-bootloader' in 'guix/scripts/system.scm'
> do, but in terms of data that can be sent with 'remote-eval'. I imagine
> the code will make more sense if you read both simultaneously.

OK, sounds good.

Some time ago, I proposed to have ‘remove-eval’ where you could do:

  (remote-eval #~(begin …) #:session …)

which would take care of building and copying everything the gexp refers
to (see
<https://lists.gnu.org/archive/html/guix-devel/2019-03/msg00127.html>.)
That would generalize a bit what you describe above.  But anyway, that’s
something that can always come later.

> Okay, on to the test suite.
>
> My understanding of the system test suite (tests run with 'check-system'
> as opposed to those run with 'check') is that the meat of the test code
> exists in a G-Expression and should _not_ be interacting with the store
> of the machine running the test suite (i.e. that's the reason we're
> using marionettes in the first place). 'gnu/tests/install.scm' seems to
> be somewhat of an exception, and because the code in '(gnu machine)'
> depends heavily on having access to a store, I've tried to extend what's
> done in 'gnu/tests/install.scm' so that my tests have access to store
> while instrumenting the marionettes.

When you say “access the store”, you mean interact with the build
daemon, right?

The (gnu tests install) tests are indeed the only ones that talk to the
build daemon.

> To be specific, the chicken and egg scenario I'm working around is that
> the SSH daemon on the marionette needs to be running in order for
> 'deploy-os' to work, but I can't call 'deploy-os' from the test
> G-Expression because the store wouldn't be accessible then.
>
> My gut is telling me that this is absolutely the wrong way to go about
> it, though. 'call-with-marionette' is one of a couple red flags making
> me feel that way -- I don't think marionettes were meant to be started
> outside the context of a derivation...

Marionettes are just a way to instrument the system under test, which is
running in a VM.

I think what you need is to ensure the target VM (the one you’re
deploying to) has a writable store.  To do that, you could create the
target VM with the equivalent of ‘guix system vm-image’, which is the
‘system-qemu-image’ procedure.  (That will require a bit of disk space,
because that image needs to be large enough to hold the store of the
machine.)

How does that sound?

> I'm going to continue to spend time on the "internals" of 'guix deploy'
> this coming week, incorporating any feedback I receive and ensuring that
> I have a framework to build upon when I extend this to more complicated
> scenarios like deploying to a VPS provider. After that, I'll tackle
> fleshing out the way deployments are specified (the file part. I'm
> holding off on the command-line tool part for right now).

Sounds good!

Consider hanging out on #guix and sending email here—it’s usually easier
to get feedback on specific topics.

Anyway, glad to see so much progress already!  Thanks for the update!

Ludo’.

Re: Progress on 'guix deploy'

[Jakob L. Kreuze]

[-- Attachment #1: Type: text/plain, Size: 1739 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> OK, sounds good.
>
> Some time ago, I proposed to have ‘remove-eval’ where you could do:
>
>   (remote-eval #~(begin …) #:session …)
>
> which would take care of building and copying everything the gexp
> refers to (see
> <https://lists.gnu.org/archive/html/guix-devel/2019-03/msg00127.html>.)
> That would generalize a bit what you describe above. But anyway,
> that’s something that can always come later.

My focus for the next few days is going to be on cleaning up the
implementation of the internals, so I'll look into that. Gaining some
knowledge of how G-Expressions work would probably serve me well in
improving my kludge of a test suite anway ;)  But I think you're right
that it wouldn't be too difficult to deal with later on.

> Marionettes are just a way to instrument the system under test, which
> is running in a VM.
>
> I think what you need is to ensure the target VM (the one you’re
> deploying to) has a writable store. To do that, you could create the
> target VM with the equivalent of ‘guix system vm-image’, which is the
> ‘system-qemu-image’ procedure. (That will require a bit of disk space,
> because that image needs to be large enough to hold the store of the
> machine.)
>
> How does that sound?

Sounds like I've got some code to read :)

It seems likely that the SSH issue I was dealing was being caused by the
store being read-only. I'll investigate that today and report back.

> Consider hanging out on #guix and sending email here—it’s usually easier to get
> feedback on specific topics.
>
> Anyway, glad to see so much progress already! Thanks for the update!

Will do!

Regards,
Jakob

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#36162] [PATCH 0/4] Add 'remote-eval'

[Jakob L. Kreuze]

[-- Attachment #1: Type: text/plain, Size: 903 bytes --]

Wow, this is great! It seems like this patch series is implementing a
number of things I'd been wishing I could just import from some other
Guix module. I'm signing off for the night, but I'll apply this tomorrow
and give you some feedback.

Ludovic Courtès <ludo@gnu.org> writes:

> There are no tests for ‘remote-eval’ currently.  It would need a VM
> with access to the store, as Jakob explained on guix-devel.

One idea I had in developing my test suite was to design another record
type to supplement '<system-test>' for tests that /do/ need access the
host's store as they're running, but thought it would be too intrusive
as it would only be used by the tests for '(gnu machine)'. Now that we
have a ‘remote-eval’ that would benefit from being tested that way,
though, I'd like to ask: is that something I should look into?

Thanks again for implementing this

Jakob

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#36162] [PATCH 0/4] Add 'remote-eval'

[Ludovic Courtès]

Hello,

zerodaysfordays@sdf.lonestar.org (Jakob L. Kreuze) skribis:

> One idea I had in developing my test suite was to design another record
> type to supplement '<system-test>' for tests that /do/ need access the
> host's store as they're running, but thought it would be too intrusive
> as it would only be used by the tests for '(gnu machine)'. Now that we
> have a ‘remote-eval’ that would benefit from being tested that way,
> though, I'd like to ask: is that something I should look into?

Tests are supposed to be deterministic, so as such, it makes sense for
tests to be normal derivations, as is currently the case.

In that spirit, we should instead tweak <virtual-machine> so that we can
instruct it to build an autonomous VM with its own store.

OTOH, we could also have ‘container-eval’, similar to ‘remote-eval’,
which would be useful for effectful code, such as code that needs to
interact with the daemon.  Food for thought!

Thanks,
Ludo’.

[bug#36162] [PATCH 0/4] Add 'remote-eval'

[Jakob L. Kreuze]

[-- Attachment #1: Type: text/plain, Size: 1063 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Tests are supposed to be deterministic, so as such, it makes sense for
> tests to be normal derivations, as is currently the case.

Makes sense. I actually hadn't realized that the Guix test suite (those
run with 'check' rather than 'check-system') were also derivations until
you mentioned it. It seems like looking at the implementation there
would give me a better idea of how to structure my tests.

> In that spirit, we should instead tweak <virtual-machine> so that we
> can instruct it to build an autonomous VM with its own store.

I like that idea. There's a 'system-qemu-image/shared-store-script', but
no 'system-qemu-image-script', so I'll have to take what I did for my
test suite experiments and turn it into something proper.

> OTOH, we could also have ‘container-eval’, similar to ‘remote-eval’,
> which would be useful for effectful code, such as code that needs to
> interact with the daemon. Food for thought!

'container' in the sense of 'guix environment' containers?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#36162] [PATCH 0/4] Add 'remote-eval'

[Ludovic Courtès]

zerodaysfordays@sdf.lonestar.org (Jakob L. Kreuze) skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> OTOH, we could also have ‘container-eval’, similar to ‘remote-eval’,
>> which would be useful for effectful code, such as code that needs to
>> interact with the daemon. Food for thought!
>
> 'container' in the sense of 'guix environment' containers?

Yes, but more generally in the sense of a process running in separate
namespaces.

Ludo’.

Re: Progress on 'guix deploy'

[Jakob L. Kreuze]

[-- Attachment #1: Type: text/plain, Size: 5709 bytes --]

zerodaysfordays@sdf.lonestar.org (Jakob L. Kreuze) writes:

> It seems likely that the SSH issue I was dealing was being caused by
> the store being read-only. I'll investigate that today and report
> back.

Sorry it took so long. It took me some significant time yesterday to
figure out how the high-level 'virtual-machine' interface actually
starts QEMU guests, but I did eventually boot a marionette with an image
produced by 'system-qemu-image' and was greeted by the same error, so I
suspect that the SSH issue isn't being caused by a read-only store.

In a Guile REPL, I created a VM image with something like the following:

#+BEGIN_SRC scheme
(use-modules ...)

(define os
  (marionette-operating-system
   (simple-operating-system
    (service dhcp-client-service-type)
    (service openssh-service-type
             (openssh-configuration
              (permit-root-login #t)
              (allow-empty-passwords? #t))))))

(with-store store
  (run-with-store store
    (mlet* %store-monad ((image (system-qemu-image os #:disk-image-size (* 3600 (expt 2 20))))
                         (_ ((store-lift build-derivations) (list image))))
      (return
       (begin
         (display (derivation->output-path image))
         (newline))))))
#+END_SRC

I proceeded to start it with 'qemu-sytem-x86_64' to get obtain the
system's store path from 'dmesg', created a writeable image with
'qemu-img' à la 'qemu-command/writable-image', and then passed the
equivalent of the following as the command for 'make-marionette':

#+BEGIN_SRC sh
export QEMU=/gnu/store/sw2rrqmjij73wcy3ajd47ypvmzh12yz6-qemu-3.1.0/
export SYSTEM=/gnu/store/rk5yy9xhxbl9zbqs78kzgxmmxd72zsq0-system/

$QEMU/bin/qemu-system-x86_64 \
    -drive file=/home/jakob/test.img,if=virtio \
    -kernel $SYSTEMkernel/bzImage \
    -initrd $SYSTEM/initrd \
    -append "console=ttyS0 --root=/dev/vda1 --system=$SYSTEM --load=$SYSTEM/boot" \
    -enable-kvm \
    -no-reboot \
    -net nic,model=virtio \
    -object rng-random,filename=/dev/urandom,id=guixsd-vm-rng \
    -device virtio-rng-pci,rng=guixsd-vm-rng \
    -vga std \
    -m 256 \
    -net user,hostfwd=tcp::2222-:22
#+END_SRC

Where '/home/jakob/test.img' is the writeable image that 'qemu-img'
produced. Running 'make check-system TESTS=ssh-deploy-os yields the same
error:

#+BEGIN_SRC
;;; (auth success)
;;; [2019/06/11 10:25:52.727456, 2] channel_open: Creating a channel 43 with 64000 window and 32768 max packet
;;; [2019/06/11 10:25:54.533529, 2] ssh_packet_global_request: Received SSH_MSG_GLOBAL_REQUEST packet
;;; [2019/06/11 10:25:54.533616, 2] ssh_packet_global_request: UNKNOWN SSH_MSG_GLOBAL_REQUEST hostkeys-00@openssh.com 0
;;; [2019/06/11 10:25:54.533638, 1] ssh_packet_process: Couldn't do anything with packet type 80
;;; [2019/06/11 10:25:54.533739, 2] ssh_packet_channel_open_conf: Received a CHANNEL_OPEN_CONFIRMATION for channel 43:0
;;; [2019/06/11 10:25:54.533775, 2] ssh_packet_channel_open_conf: Remote window : 0, maxpacket : 32768
;;; [2019/06/11 10:25:54.534340, 2] channel_rcv_change_window: Adding 2097152 bytes to channel (43:0) (from 0 bytes)
;;; [2019/06/11 10:25:54.534378, 2] channel_request: Channel request exec success
;;; [2019/06/11 10:25:54.593719, 2] grow_window: growing window (channel 43:0) to 1280000 bytes
qemu-system-x86_64: terminating on signal 15 from pid 7454 (/gnu/store/020aw068yfsq84h6scmnvfrjacmznsgz-profile/bin/guile)
;;; [2019/06/11 10:25:55.074345, 2] channel_open: Creating a channel 44 with 64000 window and 32768 max packet
;;; [2019/06/11 10:25:55.074381, 1] ssh_socket_exception_callback: Socket exception callback: 2 (-1)
;;; [2019/06/11 10:25:55.074418, 1] ssh_socket_exception_callback: Socket error: Unknown error -1
;;; [2019/06/11 10:25:55.074478, 0] channel-open-session: [GSSH ERROR] Socket error: Unknown error -1: #<input-output: channel (closed) 685d880>
Backtrace:
          15 (apply-smob/1 #<catch-closure e08a20>)
In ice-9/boot-9.scm:
    705:2 14 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
In ice-9/eval.scm:
    619:8 13 (_ #(#(#<directory (guile-user) e86140>)))
In guix/store.scm:
   623:10 12 (call-with-store _)
In guix/status.scm:
    768:4 11 (call-with-status-report _ _)
In guix/store.scm:
  1794:24 10 (run-with-store _ _ #:guile-for-build _ #:system _ # _)
In unknown file:
           9 (_ #<procedure 2df8820 at ice-9/eval.scm:330:13 ()> #<…> …)
In guix/monads.scm:
    482:9  8 (_ _)
In gnu/tests/machine.scm:
   323:12  7 (_ _)
In gnu/machine.scm:
    118:2  6 (_ _)
In guix/store.scm:
  1690:38  5 (_ #<store-connection 256.99 5364000>)
In guix/ssh.scm:
   292:18  4 (send-files #<store-connection 256.99 5364000> _ _ # _ # …)
   106:18  3 (inferior-remote-eval (begin (use-modules (guix) # …) …) …)
    99:14  2 (remote-inferior _)
In ssh/popen.scm:
     64:4  1 (open-remote-pipe _ "guix repl -t machine" "r+")
In unknown file:
           0 (channel-open-session #<input-output: channel (closed) …>)

ERROR: In procedure channel-open-session:
Throw to key `guile-ssh-error' with args `("channel-open-session" "Socket error: Unknown error -1" #<input-output: channel (closed) 685d880> #f)'.
make: *** [Makefile:5726: check-system] Error 1
#+END_SRC

I was also able to verify that the store was writeable by staring the VM
with the same parameters and running 'guix -i hello'.

Apologies for not having something more easily reproducible -- a lot of
this configuration came about by running the QEMU tools from the shell.
If desired, I can clean it up into a proper commit and push it to my
feature branch.

Regards,
Jakob

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Progress on 'guix deploy'

[Ludovic Courtès]

Hi,

zerodaysfordays@sdf.lonestar.org (Jakob L. Kreuze) skribis:

> ;;; [2019/06/11 10:25:54.534378, 2] channel_request: Channel request exec success
> ;;; [2019/06/11 10:25:54.593719, 2] grow_window: growing window (channel 43:0) to 1280000 bytes
> qemu-system-x86_64: terminating on signal 15 from pid 7454 (/gnu/store/020aw068yfsq84h6scmnvfrjacmznsgz-profile/bin/guile)
> ;;; [2019/06/11 10:25:55.074345, 2] channel_open: Creating a channel 44 with 64000 window and 32768 max packet
> ;;; [2019/06/11 10:25:55.074381, 1] ssh_socket_exception_callback: Socket exception callback: 2 (-1)
> ;;; [2019/06/11 10:25:55.074418, 1] ssh_socket_exception_callback: Socket error: Unknown error -1
> ;;; [2019/06/11 10:25:55.074478, 0] channel-open-session: [GSSH ERROR] Socket error: Unknown error -1: #<input-output: channel (closed) 685d880>

Could it be that the VM died in the middle of the SSH session?
Where does SIGTERM (15) come from?

Thanks for investigating,
Ludo’.

Re: Progress on 'guix deploy'

[Jakob L. Kreuze]

[-- Attachment #1: Type: text/plain, Size: 3771 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Could it be that the VM died in the middle of the SSH session? Where
> does SIGTERM (15) come from?

The SIGTERM comes from my less-than-ideal way of spinning up marionettes
outside of the derivation that carries out the srfi-64 tests,
'call-with-marionette'. I have a line with '(kill pid SIGTERM)' as the
QEMU guest will otherwise continue to run after the test suite exits. I
commented out the line and verified that after the error occurs, the
guest is still running (and accessible via SSH).

What's interesting is, if I leave the guest running, SSH in, set up
public key auth and run 'guix deploy' from my shell, I don't get the
socket error.

#+BEGIN_SRC
jakob@Epsilon ~/Code/guix [env] $ ./pre-inst-env guix deploy ~/Notes/Software/Guix/deployment.scm 
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
@ build-started /gnu/store/4cl8yydsdnz9rwsswrgr788l7p2q2xqy-etc.drv - x86_64-linux /var/log/guix/drvs/4c//l8yydsdnz9rwsswrgr788l7p2q2xqy-etc.drv.bz2 23724
@ build-succeeded /gnu/store/4cl8yydsdnz9rwsswrgr788l7p2q2xqy-etc.drv -
@ build-started /gnu/store/ban0c1b97gfzy5siqwzd30j3m31slxfq-activate-service.drv - x86_64-linux /var/log/guix/drvs/ba//n0c1b97gfzy5siqwzd30j3m31slxfq-activate-service.drv.bz2 23732
@ build-succeeded /gnu/store/ban0c1b97gfzy5siqwzd30j3m31slxfq-activate-service.drv -
@ build-started /gnu/store/8wadw74kz2nwx6n7gmilkjn4qig3isrn-activate.drv - x86_64-linux /var/log/guix/drvs/8w//adw74kz2nwx6n7gmilkjn4qig3isrn-activate.drv.bz2 23739
@ build-succeeded /gnu/store/8wadw74kz2nwx6n7gmilkjn4qig3isrn-activate.drv -
@ build-started /gnu/store/yz4brqis8jfs08wjhyk1bbdcz6vdx2ld-boot.drv - x86_64-linux /var/log/guix/drvs/yz//4brqis8jfs08wjhyk1bbdcz6vdx2ld-boot.drv.bz2 23745
@ build-succeeded /gnu/store/yz4brqis8jfs08wjhyk1bbdcz6vdx2ld-boot.drv -
@ build-started /gnu/store/dqxm5lw45408f64lqm5k2ylyx7vvqpyw-system.drv - x86_64-linux /var/log/guix/drvs/dq//xm5lw45408f64lqm5k2ylyx7vvqpyw-system.drv.bz2 23751
@ build-succeeded /gnu/store/dqxm5lw45408f64lqm5k2ylyx7vvqpyw-system.drv -
building gnu-deployed... done
sending 4790 store items (1,905 MiB) to 'localhost'...
Backtrace:
          10 (primitive-load "/home/jakob/.config/guix/current/bin/g…")
In guix/ui.scm:
  1747:12  9 (run-guix-command _ . _)
In guix/store.scm:
   623:10  8 (call-with-store _)
In srfi/srfi-1.scm:
    640:9  7 (for-each #<procedure 5088840 at guix/scripts/deploy.s…> …)
In guix/scripts/deploy.scm:
    84:20  6 (_ _)
In guix/store.scm:
  1794:24  5 (run-with-store _ _ #:guile-for-build _ #:system _ # _)
In gnu/machine.scm:
    119:2  4 (_ _)
In guix/store.scm:
  1690:38  3 (_ #<store-connection 256.99 3187000>)
In guix/ssh.scm:
    313:4  2 (send-files _ _ _ #:recursive? _ #:log-port _)
In guix/store.scm:
  1562:12  1 (export-paths #<store-connection 256.99 3187000> _ #<i…> …)
  1542:22  0 (export-path #<store-connection 256.99 3187000> _ #<in…> …)

guix/store.scm:1542:22: In procedure export-path:
Throw to key `srfi-34' with args `(#<condition &store-protocol-error [message: "hash of path `/gnu/store/ag8ilfwcnxn0xnw8bar9ia73h3hg3rra-gs-fonts-8.11' has changed from `67ae533629b1f24d9736c6daf5f6439e5cfce18c7fa2f1c6d45e65bbf0c471b8' to `dd9339c8d709b867f3ff056b2f2e1aa368cfa2312302fc30392d820c2f3ef9ab'!" status: 1] 8093d20>)'.
#+END_SRC

Very unusal. I'm wondering if it's caused by my use of
'call-with-connected-session/auth' from (gnu tests ssh) rather than
'open-ssh-session' in (guix ssh). That's the main difference between the
test suite and 'guix deploy' in terms of how they communicate with the
guest.

Regards,
Jakob

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Progress on 'guix deploy'

[Jakob L. Kreuze]

[-- Attachment #1: Type: text/plain, Size: 4970 bytes --]

zerodaysfordays@sdf.lonestar.org (Jakob L. Kreuze) writes:

> Very unusal. I'm wondering if it's caused by my use of
> 'call-with-connected-session/auth' from (gnu tests ssh) rather than
> 'open-ssh-session' in (guix ssh). That's the main difference between
> the test suite and 'guix deploy' in terms of how they communicate with
> the guest.

The "[w]e need lightweight compression when exchanging full archives"
comment in (guix ssh) gave me a little bit of hope that the issue could
be solved by addressing the difference in how SSH sessions were being
opened, but I've ensured that they're opened with the same parameters
and still run into this seemingly nondeterministic behavior. Well, I
suppose it's deterministic in the sense that it always fails when run
from the test suite, and always works when run from 'guix deploy' ;)

Running both with `#:log-verbosity 'protocol` yields essentially the
same output, aside from the authentication method.

#+BEGIN_SRC
diff -u /home/jakob/ssh-clean-2 /home/jakob/ssh-clean-1
--- /home/jakob/ssh-clean-2	2019-06-12 12:25:33.856336431 -0400
+++ /home/jakob/ssh-clean-1	2019-06-12 12:25:03.056337396 -0400
@@ -1,5 +1,9 @@
+1] ssh_config_parse_line: Unsupported option: IdentitiesOnly, line: 4
+1] ssh_config_parse_line: Unsupported option: IdentitiesOnly, line: 9
+1] ssh_config_parse_line: Unsupported option: IdentitiesOnly, line: 14
+1] ssh_config_parse_line: Unsupported option: IdentitiesOnly, line: 19
 2] ssh_connect: libssh 0.8.7 (c) 2003-2018 Aris Adamantiadis, Andreas Schneider and libssh contributors. Distributed under the LGPL, please refer to COPYING file for information about your rights, using threading threads_pthread
-2] ssh_socket_connect: Nonblocking connection socket: 11
+2] ssh_socket_connect: Nonblocking connection socket: 16
 2] ssh_connect: Socket connecting, now waiting for the callbacks to work
 1] socket_callback_connected: Socket connection callback: 1 (0)
 1] ssh_client_connection_callback: SSH server banner: SSH-2.0-OpenSSH_8.0
@@ -11,16 +15,34 @@
 2] ssh_client_curve25519_reply: SSH_MSG_NEWKEYS sent
 2] ssh_packet_newkeys: Received SSH_MSG_NEWKEYS
 2] ssh_packet_newkeys: Signature verified and valid
+2] ssh_pki_import_privkey_base64: Trying to decode privkey passphrase=false
+2] ssh_pki_openssh_import: Opening OpenSSH private key: ciphername: none, kdf: none, nkeys: 1
+2] ssh_userauth_publickey_auto: Successfully authenticated using /home/jakob/.ssh/guixsd_slave
 2] channel_open: Creating a channel 43 with 64000 window and 32768 max packet
 2] ssh_packet_global_request: Received SSH_MSG_GLOBAL_REQUEST packet
 2] ssh_packet_global_request: UNKNOWN SSH_MSG_GLOBAL_REQUEST hostkeys-00@openssh.com 0
 1] ssh_packet_process: Couldn't do anything with packet type 80
+2] ssh_packet_ignore_callback: Received SSH_MSG_DEBUG packet
+2] ssh_packet_ignore_callback: Received SSH_MSG_DEBUG packet
 2] ssh_packet_channel_open_conf: Received a CHANNEL_OPEN_CONFIRMATION for channel 43:0
 2] ssh_packet_channel_open_conf: Remote window : 0, maxpacket : 32768
 2] channel_rcv_change_window: Adding 2097152 bytes to channel (43:0) (from 0 bytes)
 2] channel_request: Channel request exec success
 2] grow_window: growing window (channel 43:0) to 1280000 bytes
 2] channel_open: Creating a channel 44 with 64000 window and 32768 max packet
-1] ssh_socket_exception_callback: Socket exception callback: 2 (-1)
-1] ssh_socket_exception_callback: Socket error: Unknown error -1
-0] channel-open-session: [GSSH ERROR] Socket error: Unknown error -1: #<input-output: channel (closed) 6c024c0>
+2] ssh_packet_channel_open_conf: Received a CHANNEL_OPEN_CONFIRMATION for channel 44:1
+2] ssh_packet_channel_open_conf: Remote window : 0, maxpacket : 32768
+2] channel_rcv_change_window: Adding 2097152 bytes to channel (44:1) (from 0 bytes)
+2] channel_request: Channel request exec success
+2] grow_window: growing window (channel 44:1) to 1280000 bytes
+2] channel_rcv_change_window: Adding 112751 bytes to channel (44:1) (from 1939345 bytes)
+2] channel_rcv_change_window: Adding 96256 bytes to channel (44:1) (from 1961984 bytes)
+2] channel_rcv_change_window: Adding 4096 bytes to channel (44:1) (from 1913495 bytes)
+2] channel_rcv_change_window: Adding 65536 bytes to channel (44:1) (from 1917591 bytes)
+2] channel_rcv_change_window: Adding 65536 bytes to channel (44:1) (from 1983127 bytes)
+2] channel_open: Creating a channel 45 with 64000 window and 32768 max packet
+2] ssh_packet_channel_open_conf: Received a CHANNEL_OPEN_CONFIRMATION for channel 45:1
+2] ssh_packet_channel_open_conf: Remote window : 0, maxpacket : 32768
+2] channel_rcv_change_window: Adding 2097152 bytes to channel (45:1) (from 0 bytes)
+2] channel_request: Channel request exec success
+2] channel_rcv_change_window: Adding 129520 bytes to channel (45:1) (from 1902096 bytes)

...

Diff finished.  Wed Jun 12 12:25:42 2019
#+END_SRC

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Progress on 'guix deploy'

[Ludovic Courtès]

Hi Jakob,

zerodaysfordays@sdf.lonestar.org (Jakob L. Kreuze) skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> OK, sounds good.
>>
>> Some time ago, I proposed to have ‘remove-eval’ where you could do:
>>
>>   (remote-eval #~(begin …) #:session …)
>>
>> which would take care of building and copying everything the gexp
>> refers to (see
>> <https://lists.gnu.org/archive/html/guix-devel/2019-03/msg00127.html>.)
>> That would generalize a bit what you describe above. But anyway,
>> that’s something that can always come later.
>
> My focus for the next few days is going to be on cleaning up the
> implementation of the internals, so I'll look into that.

I finally got around to working on it:

  https://issues.guix.gnu.org/issue/36162

Let me know what you think!

Ludo’.

Re: Progress on 'guix deploy'

[Jakob L. Kreuze]

[-- Attachment #1: Type: text/plain, Size: 787 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> I finally got around to working on it:
>
>   https://issues.guix.gnu.org/issue/36162
>
> Let me know what you think!

Thanks for implementing it :)  Not sure if you saw, but I sent two
emails regarding the patch series to guix-patches. They're down at the
bottom of that link you attached.

My thoughts, chiefly, are that it looks very promising, but seems to
bring a few issues with other G-Expression objects. Applying the patch
series to master causes the 'channel-instances->manifest' and
'make-lzip-input-port/compressed' tests to fail. I may end up looking
into it further this evening, but at the moment, I'm not sure I'd know
how to go about debugging something that's being run in the store monad.

Regards,
Jakob

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]