[bug#36162] [PATCH 0/4] Add 'remote-eval'

[bug#36162] [PATCH 0/4] Add 'remote-eval'

[Ludovic Courtès]

Hello Guix!

This patch series add ‘remote-eval’, which takes a gexp, remotely deploys
everything the gexp refers to, and evaluates it (see
<https://lists.gnu.org/archive/html/guix-devel/2019-03/msg00127.html> for
the initial discussion.)  So you can have gexps like:

  #~(execl #$(file-append ffmpeg "/bin/ffmpeg") …)

When you evaluate it, this specific ‘ffmpeg’ will be deployed over there.
Another example is:

  (with-imported-modules (source-module-closure '((gnu services herd)))
    #~(begin
        (use-modules (gnu services herd))
        (map live-service-provision (current-services))))

This gexp, when evaluated remotely, will use your very own (gnu services
herd) module and the corresponding Guile (so if you’re on Guile 3 and the
remote is still on Guile 2, that’s fine: Guile 3 will first be deployed
there.)

‘remote-eval’ allows you to build locally and send the build results,
or to send the derivations and build remotely.

The use case is for code that deals with state or has a side effect.
Otherwise you’d just use a derivation and offload it.

There are no tests for ‘remote-eval’ currently.  It would need a VM
with access to the store, as Jakob explained on guix-devel.

Thoughts?

Ludo’.

Ludovic Courtès (4):
  gexp: Add 'lower-gexp' and express 'gexp->derivation' in terms of it.
  Add (guix repl).
  inferior: Add 'read-repl-response'.
  Add (guix remote).

 Makefile.am           |   2 +
 guix/gexp.scm         | 238 +++++++++++++++++++++++++++++++-----------
 guix/inferior.scm     |   9 +-
 guix/remote.scm       | 130 +++++++++++++++++++++++
 guix/repl.scm         |  86 +++++++++++++++
 guix/scripts/repl.scm |  56 +---------
 tests/gexp.scm        |  37 +++++++
 7 files changed, 444 insertions(+), 114 deletions(-)
 create mode 100644 guix/remote.scm
 create mode 100644 guix/repl.scm

-- 
2.21.0

[bug#36162] [PATCH 0/4] Add 'remote-eval'

[Jakob L. Kreuze]

[-- Attachment #1: Type: text/plain, Size: 903 bytes --]

Wow, this is great! It seems like this patch series is implementing a
number of things I'd been wishing I could just import from some other
Guix module. I'm signing off for the night, but I'll apply this tomorrow
and give you some feedback.

Ludovic Courtès <ludo@gnu.org> writes:

> There are no tests for ‘remote-eval’ currently.  It would need a VM
> with access to the store, as Jakob explained on guix-devel.

One idea I had in developing my test suite was to design another record
type to supplement '<system-test>' for tests that /do/ need access the
host's store as they're running, but thought it would be too intrusive
as it would only be used by the tests for '(gnu machine)'. Now that we
have a ‘remote-eval’ that would benefit from being tested that way,
though, I'd like to ask: is that something I should look into?

Thanks again for implementing this

Jakob

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#36162] [PATCH 0/4] Add 'remote-eval'

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> This patch series add ‘remote-eval’, which takes a gexp, remotely deploys
> everything the gexp refers to, and evaluates it (see
> <https://lists.gnu.org/archive/html/guix-devel/2019-03/msg00127.html> for
> the initial discussion.)  So you can have gexps like:
>
>   #~(execl #$(file-append ffmpeg "/bin/ffmpeg") …)
>
> When you evaluate it, this specific ‘ffmpeg’ will be deployed over there.
> Another example is:
>
>   (with-imported-modules (source-module-closure '((gnu services herd)))
>     #~(begin
>         (use-modules (gnu services herd))
>         (map live-service-provision (current-services))))

> This gexp, when evaluated remotely, will use your very own (gnu services
> herd) module and the corresponding Guile (so if you’re on Guile 3 and the
> remote is still on Guile 2, that’s fine: Guile 3 will first be deployed
> there.)
>
> ‘remote-eval’ allows you to build locally and send the build results,
> or to send the derivations and build remotely.

This is great and just what I need for the install-berlin.scm script in
the “maintenance” repository where I need to deploy a particular version
of Guix to the target system before using that version of Guix to
reconfigure the remote system.

Thank you!

--
Ricardo

[bug#36162] [PATCH 0/4] Add 'remote-eval'

[Jakob L. Kreuze]

[-- Attachment #1: Type: text/plain, Size: 2052 bytes --]

zerodaysfordays@sdf.lonestar.org (Jakob L. Kreuze) writes:

> Wow, this is great! It seems like this patch series is implementing a
> number of things I'd been wishing I could just import from some other
> Guix module. I'm signing off for the night, but I'll apply this tomorrow
> and give you some feedback.

It seems this breaks 'operating-system-derivation'.

#+BEGIN_SRC
Backtrace:
          19 (_ #<procedure 40d32e0 at ice-9/eval.scm:330:13 ()> #<…> …)
In gnu/services.scm:
    320:2 18 (_ _)
In gnu/system.scm:
    461:4 17 (_ _)
In unknown file:
          16 (_ #<procedure 40d9220 at ice-9/eval.scm:330:13 ()> #<…> …)
          15 (_ #<procedure 2ebc160 at ice-9/eval.scm:330:13 ()> #<…> …)
In guix/monads.scm:
    482:9 14 (_ _)
In unknown file:
          13 (_ #<procedure 2ebc100 at ice-9/eval.scm:330:13 ()> #<…> …)
In guix/store.scm:
   1667:8 12 (_ _)
   1667:8 11 (_ _)
In unknown file:
          10 (_ #<procedure 2f1d940 at ice-9/eval.scm:330:13 ()> #<…> …)
           9 (_ #<procedure 2f2c460 at ice-9/eval.scm:330:13 ()> #<…> …)
In guix/monads.scm:
    482:9  8 (_ _)
In unknown file:
           7 (_ #<procedure 26ba500 at ice-9/eval.scm:330:13 ()> #<…> …)
In guix/store.scm:
   1667:8  6 (_ _)
In unknown file:
           5 (_ #<procedure 2701740 at ice-9/eval.scm:330:13 ()> #<…> …)
           4 (_ #<procedure 27b2200 at ice-9/eval.scm:330:13 ()> #<…> …)
In ice-9/eval.scm:
   191:27  3 (_ #(#(#<directory (guix gexp) c8f640> #<procedure…>) …))
   173:47  2 (_ #(#(#<directory (guix gexp) c8f640> "module-im…" …) …))
    159:9  1 (_ #(#(#<directory (guix gexp) c8f640> "module-im…" …) …))
In guix/derivations.scm:
   597:28  0 (derivation->output-path #f _)

guix/derivations.scm:597:28: In procedure derivation->output-path:
In procedure struct_vtable: Wrong type argument in position 1 (expecting struct): #f
building gnu-deployed...
#+END_SRC

'git bisect' seems to think that 9dddee345b introduced it.

Regards,
Jakob

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#36162] [PATCH 0/4] Add 'remote-eval'

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 641 bytes --]

zerodaysfordays@sdf.lonestar.org (Jakob L. Kreuze) skribis:

> zerodaysfordays@sdf.lonestar.org (Jakob L. Kreuze) writes:
>
>> Wow, this is great! It seems like this patch series is implementing a
>> number of things I'd been wishing I could just import from some other
>> Guix module. I'm signing off for the night, but I'll apply this tomorrow
>> and give you some feedback.
>
> It seems this breaks 'operating-system-derivation'.

[...]

> In guix/derivations.scm:
>    597:28  0 (derivation->output-path #f _)

Oops!  The patch below fixes it.

Let me know if you find other issues.  Thanks for testing!

Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1198 bytes --]

diff --git a/guix/gexp.scm b/guix/gexp.scm
index 38f64db7f1..ab29c2494e 100644
--- a/guix/gexp.scm
+++ b/guix/gexp.scm
@@ -720,7 +720,7 @@ derivations--e.g., code evaluated for its side effects."
                                                        #:system system
                                                        #:module-path module-path
                                                        #:extensions extensions
-                                                       #:guile guile-for-build
+                                                       #:guile guile
                                                        #:pre-load-modules?
                                                        pre-load-modules?
                                                        #:deprecation-warnings
@@ -746,7 +746,7 @@ derivations--e.g., code evaluated for its side effects."
                                     '())
                               ,@(map gexp-input exts)
                               ,@inputs)
-                            guile-for-build
+                            guile
                             load-path
                             load-compiled-path)))))
 

[bug#36162] [PATCH 0/4] Add 'remote-eval'

[Ludovic Courtès]

Hello,

zerodaysfordays@sdf.lonestar.org (Jakob L. Kreuze) skribis:

> One idea I had in developing my test suite was to design another record
> type to supplement '<system-test>' for tests that /do/ need access the
> host's store as they're running, but thought it would be too intrusive
> as it would only be used by the tests for '(gnu machine)'. Now that we
> have a ‘remote-eval’ that would benefit from being tested that way,
> though, I'd like to ask: is that something I should look into?

Tests are supposed to be deterministic, so as such, it makes sense for
tests to be normal derivations, as is currently the case.

In that spirit, we should instead tweak <virtual-machine> so that we can
instruct it to build an autonomous VM with its own store.

OTOH, we could also have ‘container-eval’, similar to ‘remote-eval’,
which would be useful for effectful code, such as code that needs to
interact with the daemon.  Food for thought!

Thanks,
Ludo’.

[bug#36162] [PATCH 0/4] Add 'remote-eval'

[Jakob L. Kreuze]

[-- Attachment #1: Type: text/plain, Size: 1869 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Let me know if you find other issues. Thanks for testing!

This fixes the test suite, but I'm having an issue with 'remote-eval'
now. A G-Expression with no inputs such as

#+BEGIN_SRC scheme
(mlet %store-monad ((result (remote-eval #~(begin (string-append "Hello, " "world!"))
                             (ssh-session machine))))
  (display result)
  (newline))
#+END_SRC

results in the following error:

#+BEGIN_SRC
Backtrace:
          13 (apply-smob/1 #<catch-closure 1954a20>)
In ice-9/boot-9.scm:
    705:2 12 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
In ice-9/eval.scm:
    619:8 11 (_ #(#(#<directory (guile-user) 19d2140>)))
In guix/ui.scm:
  1747:12 10 (run-guix-command _ . _)
In guix/store.scm:
   623:10  9 (call-with-store _)
In srfi/srfi-1.scm:
    640:9  8 (for-each #<procedure 5fe5600 at guix/scripts/deploy.s…> …)
In guix/scripts/deploy.scm:
    84:20  7 (_ _)
In guix/store.scm:
  1794:24  6 (run-with-store _ _ #:guile-for-build _ #:system _ # _)
In unknown file:
           5 (_ #<procedure 5ff43c0 at ice-9/eval.scm:330:13 ()> #<…> …)
In guix/remote.scm:
   116:10  4 (_ _)
In guix/store.scm:
  1690:38  3 (_ #<store-connection 256.99 5da0960>)
In guix/derivations.scm:
   987:22  2 (build-derivations #<store-connection 256.99 5da0960> # _)
In srfi/srfi-1.scm:
   592:17  1 (map1 ((#<derivation /gnu/store/16c8c8hm1qdn6xz…> . #) …))
In guix/derivations.scm:
   987:27  0 (_ _)

guix/derivations.scm:987:27: Throw to key `match-error' with args `("match" "no matching pattern" (#<derivation /gnu/store/16c8c8hm1qdn6xz8014939mirc7c4d4j-guile-2.2.4.drv => /gnu/store/cz91c7nvg7495vacv44bxmqnn8sha3cb-guile-2.2.4-debug /gnu/store/r658y3cgpnf99nxjxqgjiaizx20ac4k0-guile-2.2.4 47e69b0> . "out"))'.
#+END_SRC

Regards,
Jakob

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#36162] [PATCH 0/4] Add 'remote-eval'

[Jakob L. Kreuze]

[-- Attachment #1: Type: text/plain, Size: 1063 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Tests are supposed to be deterministic, so as such, it makes sense for
> tests to be normal derivations, as is currently the case.

Makes sense. I actually hadn't realized that the Guix test suite (those
run with 'check' rather than 'check-system') were also derivations until
you mentioned it. It seems like looking at the implementation there
would give me a better idea of how to structure my tests.

> In that spirit, we should instead tweak <virtual-machine> so that we
> can instruct it to build an autonomous VM with its own store.

I like that idea. There's a 'system-qemu-image/shared-store-script', but
no 'system-qemu-image-script', so I'll have to take what I did for my
test suite experiments and turn it into something proper.

> OTOH, we could also have ‘container-eval’, similar to ‘remote-eval’,
> which would be useful for effectful code, such as code that needs to
> interact with the daemon. Food for thought!

'container' in the sense of 'guix environment' containers?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#36162] [PATCH 0/4] Add 'remote-eval'

[Ludovic Courtès]

zerodaysfordays@sdf.lonestar.org (Jakob L. Kreuze) skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> OTOH, we could also have ‘container-eval’, similar to ‘remote-eval’,
>> which would be useful for effectful code, such as code that needs to
>> interact with the daemon. Food for thought!
>
> 'container' in the sense of 'guix environment' containers?

Yes, but more generally in the sense of a process running in separate
namespaces.

Ludo’.

[bug#36162] [PATCH 0/4] Add 'remote-eval'

[Ludovic Courtès]

Hi,

zerodaysfordays@sdf.lonestar.org (Jakob L. Kreuze) skribis:

> In guix/derivations.scm:
>    987:22  2 (build-derivations #<store-connection 256.99 5da0960> # _)
> In srfi/srfi-1.scm:
>    592:17  1 (map1 ((#<derivation /gnu/store/16c8c8hm1qdn6xz…> . #) …))
> In guix/derivations.scm:
>    987:27  0 (_ _)
>
> guix/derivations.scm:987:27: Throw to key `match-error' with args `("match" "no matching pattern" (#<derivation /gnu/store/16c8c8hm1qdn6xz8014939mirc7c4d4j-guile-2.2.4.drv => /gnu/store/cz91c7nvg7495vacv44bxmqnn8sha3cb-guile-2.2.4-debug /gnu/store/r658y3cgpnf99nxjxqgjiaizx20ac4k0-guile-2.2.4 47e69b0> . "out"))'.

I think that’s because you’re on an older Guix, which lacks commit
f8a9f99cd602ce1dc5307cb0c21ae718ad8796bb.

HTH!

Ludo’.

[bug#36162] [PATCH 0/4] Add 'remote-eval'

[Jakob L. Kreuze]

[-- Attachment #1: Type: text/plain, Size: 353 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> I think that’s because you’re on an older Guix, which lacks commit
> f8a9f99cd602ce1dc5307cb0c21ae718ad8796bb.

Guess I let my feature branch go a little stale. Fetched from upstream
and it works now :)  I'll have a go at reimplementing 'deploy-os' and
let you know if I run into any other issues.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#36162] [PATCH 0/4] Add 'remote-eval'

[Jakob L. Kreuze]

[-- Attachment #1: Type: text/plain, Size: 592 bytes --]

zerodaysfordays@sdf.lonestar.org (Jakob L. Kreuze) writes:

> Guess I let my feature branch go a little stale. Fetched from upstream
> and it works now :) I'll have a go at reimplementing 'deploy-os' and
> let you know if I run into any other issues.

Works like a charm! There's more code for me to clean up, but we've got
a preliminary reimplementation of 'deploy-os' using your 'remote-eval'.
Pushed it to my personal branch if you'd like to take a look.

  https://git.sr.ht/~jakob/guix/commit/ca8e7b77cef4a60e9d820d6ae05b8b59f64b4f23

Thanks again for implementing this!

Regards,
Jakob

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

[bug#36162] [PATCH 0/4] Add 'remote-eval'

[Ludovic Courtès]

Hi Jakob,

zerodaysfordays@sdf.lonestar.org (Jakob L. Kreuze) skribis:

> zerodaysfordays@sdf.lonestar.org (Jakob L. Kreuze) writes:
>
>> Guess I let my feature branch go a little stale. Fetched from upstream
>> and it works now :) I'll have a go at reimplementing 'deploy-os' and
>> let you know if I run into any other issues.
>
> Works like a charm! There's more code for me to clean up, but we've got
> a preliminary reimplementation of 'deploy-os' using your 'remote-eval'.
> Pushed it to my personal branch if you'd like to take a look.
>
>   https://git.sr.ht/~jakob/guix/commit/ca8e7b77cef4a60e9d820d6ae05b8b59f64b4f23

Woow, that was fast!  It looks good, and the patch removes a few lines
of code, which is always nice.  :-)

I’ll push the ‘remote-eval’ things shortly.  There’s certainly a few
things we can improve, and one of them is progress report: there’s
currently no call to ‘show-what-to-build’, and that’s obviously not
great.

Thank you for the great feedback!

Ludo’.

[bug#36162] [PATCH 0/4] Add 'remote-eval'

[Christopher Lemmer Webber]

Ludovic Courtès writes:

> I’ll push the ‘remote-eval’ things shortly.  There’s certainly a few
> things we can improve, and one of them is progress report: there’s
> currently no call to ‘show-what-to-build’, and that’s obviously not
> great.

Hi Ludo'!

It looks like Jakob's code is nearing the point where we can merge it
into Guix proper.  I'd like to get in the "guix deploy" stuff as soon as
it's good enough to merge.  Would you mind pushing up your remote-eval
stuff, if you're confident enough to do so, since it's a preliminary
requirement for "guix deploy"?

[bug#36162] [PATCH 0/4] Add 'remote-eval'

[Ludovic Courtès]

Hello!

Christopher Lemmer Webber <cwebber@dustycloud.org> skribis:

> It looks like Jakob's code is nearing the point where we can merge it
> into Guix proper.  I'd like to get in the "guix deploy" stuff as soon as
> it's good enough to merge.  Would you mind pushing up your remote-eval
> stuff, if you're confident enough to do so, since it's a preliminary
> requirement for "guix deploy"?

I finally went ahead and pushed it, after adding a note about the
instability of the API.  :-)

I think ‘remote-eval’ shouldn’t change much, but I’m not fully satisfied
with ‘lower-gexp’ in that it’s a bit too redundant with <derivation>
etc.  I couldn’t turn that vague feeling into actual code changes
though, so I’ll keep looking at it in the background, and it shouldn’t
have any impact on how ‘guix deploy’ uses ‘remote-eval’.

Apologies for the delay!

Ludo’.