Re: building "guix deploy"

[Christopher Lemmer Webber <cwebber@dustycloud.org>]

Ludovic Courtès writes:

> Hi there!

Hi Ludo!  Thanks for your reply.

> "Thompson, David" <dthompson2@worcester.edu> skribis:
>
>> Chris Webber and I spent the morning chatting about how we want to
>> approach making "guix deploy" a reality and then started hacking on it
>> in the afternoon.  Although we weren't able to complete a working
>> prototype by the end of the day, we were able to get pretty close.  We
>> created a 'guix deploy' CLI to build derivations for any number of
>> remote systems on a local workstation and initiate the transfer to the
>> remote systems, but we encountered a difficult to debug SSH error that
>> blocked our progress:
>>
>> sending 85 store items (0 MiB) to 'test.activitypub.rocks'...
>> exporting path `/gnu/store/ylcdmrj3vf00ixdcjvkl3mbs8f5i9w8l-git-minimal-2.20.1.drv'
>> ;;; [2019/03/09 17:32:48.792589, 0] write_to_channel_port: [GSSH
>> ERROR] Remote channel is closed: #<input-output: channel (open)
>> 541a220>
>> Backtrace:
>>           10 (apply-smob/1 #<catch-closure a26900>)
>> In ice-9/boot-9.scm:
>>     705:2  9 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
>> In ice-9/eval.scm:
>>     619:8  8 (_ #(#(#<directory (guile-user) ab1140>)))
>> In guix/ui.scm:
>>   1654:12  7 (run-guix-command _ . _)
>> In guix/scripts/deploy.scm:
>>      72:8  6 (guix-deploy . _)
>> In srfi/srfi-1.scm:
>>     640:9  5 (for-each #<procedure 4e5c940 at guix/scripts/deploy.s…> …)
>> In guix/scripts/deploy.scm:
>>     74:20  4 (_ _)
>> In gnu/machine.scm:
>>     58:22  3 (_ _ _ _)
>> In guix/ssh.scm:
>>     313:4  2 (send-files #<store-connection 256.99 48000f0> _ _ # _ # …)
>> In guix/store.scm:
>>    1504:7  1 (export-paths #<store-connection 256.99 48000f0> _ #<i…> …)
>> In unknown file:
>>            0 (put-bytevector #<input-output: channel (open) 541a220> …)
>>
>> ERROR: In procedure put-bytevector:
>> Throw to key `guile-ssh-error' with args `("write_to_channel_port"
>> "Remote channel is closed" #<input-output: channel (open) 541a220>
>> #f)'.
>>
>> If anyone knows what might be going on here and how we could resolve
>> it, your input would be much appreciated!  We verified via the sshd
>> logs that we were indeed successfully establishing a connection.
>
> Error reporting in (guix ssh) is, ahem, not as good as it could be.
>
> Apparently the SSH channel was closed prematurely, which could be due to
> a number of things:
>
>   1. Are ‘guix’ and ‘guile’ in $PATH on the remote machine, for
>      non-interactive shells?
>
>   2. Is ‘guix repl’ available in the remote machine?
>
> You can test this with:
>
>   ssh HOST guile --version
>   ssh HOST guix repl --version

Yep, both respond with
  guile (GNU Guile) 2.2.4
and
  guix (GNU Guix) 0.16.0-10.2637cfd
respectively.

> Also, does ‘guix copy’ fail similarly when sending files to that host?

It seems it does:

cwebber@jasmine:~/devel/librelounge-audio$ guix copy --to=test.activitypub.rocks pidgin
guile: warning: failed to install locale
sending 37 store items (336 MiB) to 'test.activitypub.rocks'...
;;; [2019/03/11 10:39:25.573104, 0] write_to_channel_port: [GSSH ERROR] Remote channel is closed: #<input-output: channel (open) 46f5e60>
Backtrace:
          11 (primitive-load "/home/cwebber/.config/guix/current/bin…")
In guix/ui.scm:
  1654:12 10 (run-guix-command _ . _)
In ice-9/boot-9.scm:
    829:9  9 (catch _ _ #<procedure 7fca74f959b8 at guix/ui.scm:624…> …)
    829:9  8 (catch _ _ #<procedure 7fca74f959d0 at guix/ui.scm:750…> …)
In guix/status.scm:
    810:4  7 (call-with-status-report _ _)
In guix/scripts/copy.scm:
    81:27  6 (send-to-remote-host _ _)
In guix/ssh.scm:
    313:4  5 (send-files #<store-connection 256.97 27a5eb0> _ _ # _ # …)
In guix/store.scm:
  1505:12  4 (export-paths #<store-connection 256.97 27a5eb0> _ #<i…> …)
  1485:22  3 (export-path #<store-connection 256.97 27a5eb0> _ #<in…> …)
   683:13  2 (process-stderr _ _)
   646:10  1 (dump-port #<input-output: socket 14> #<input-output: …> …)
In unknown file:
           0 (put-bytevector #<input-output: channel (open) 46f5e60> …)

ERROR: In procedure put-bytevector:
Throw to key `guile-ssh-error' with args `("write_to_channel_port" "Remote channel is closed" #<input-output: channel (open) 46f5e60> #f)'.

I wonder what got screwed up!

>> Once we're past this blocking issue and are able to transfer OS
>> closures to remote systems, we plan to write a modified version of
>> switch-to-system that uses guile-ssh to switch remote symlinks for the
>> active system and run the activation script.  We'll save
>> upgrade-shepherd-services for later, as it is quite a bit more
>> complex.
>
> My plan is to have ‘guix system reconfigure --host=host.example.org’.
> To do that, I thought about the following plan:
>
>   1. Isolate the effectful part of reconfigure (basically
>      ‘switch-to-system’).
>
>   2. Implement ‘remote-eval’, which takes a gexp and an SSH session and
>      evaluates the expression remotely, copying the gexp inputs as
>      needed.
>
>   3. Have ‘reconfigure’ use either ‘eval’ or ‘remote-eval’ to evaluate
>      the effectful bits of reconfigure.

This sounds like the right approach to me.

> #1 is a bit annoying because we need to untangle code so that we can
> easily put it all “on the build side.”  In particular, I think we’ll
> have to change (guix graph), used by ‘upgrade-shepherd-services’, so
> that it no longer depends on ‘%store-monad’.
>
> That said, it’s probably a good idea to take a shorter path in the
> meantime to unlock progress on ‘guix deploy’!
>
>> There's not a lot of code yet, but you can check it out in the
>> wip-deploy2 branch.  Currently, the only supported use-case is running
>> the equivalent of 'guix system reconfigure' on machines already
>> running GuixSD that have an OpenSSH daemon running, but the basic
>> design allows for additional use-cases to be supported in the future.
>
> Yay!
>
> Thank you gentlefolks for resuming work on this!
>
> Ludo’.

Thank you for your response!

By the way, since the problem somehow seems to be a problem with my
server and ssh + guile-ssh (!!!) I wonder if someone else has a server
set up they'd be brave enough to try the above branch with?  It appears
that the problem is not Dave's code, given that "guix copy" also fails.