Viewing derivation output in the store

Viewing derivation output in the store

[Brian Cully]

	I apologize if this isn’t the right place to ask these
questions. While this is on the level of beginner tutorial (or should
be, IMHO), the devel list seemed the most relevant.

	I’m trying to figure out how I can create a file and view it in
the store before installation. I’d like to make sure the target looks
the way I want before putting it in its final location. I would like to
do this entirely within a Geiser session, rather than having to call
shell commands for ease of edit-test-debug cycles and tracing.

	This is also for pedagogical purposes; I’m trying to learn how
Guix goes from abstract file-like-object to an actual file. I’m still
new to basically everything here, and exercises like this help me learn.

	So, given the following:

---[snip]---
(use-modules (guix)
             (guix gexp)
             (guix monads)
             (gnu))

(define test-file (plain-file "foo.txt" "the contents of foo"))
(define test-gexp #~(begin #$test-file))
(define test-drv (gexp->derivation "foo" test-gexp))
---[snip]---

	How do I go from the derivation in ’test-drv’ to the output in
the store? Something like how, from the REPL, this works:

---[snip]---
scheme@(guile-user)> ,enter-store-monad
store-monad@(guile-user) [1]> (text-file "foo" "hello")
$34 = "/gnu/store/mnzh1q6ilbw0bg04dg0vc4f47laz57lg-foo"
---[snip]---

	Now I can easily open the path to ’/gnu/store/…foo’ to verify
the contents are correct.

NB: While I’m using ’plain-file’ here and its conents are obvious, this
is just to keep things simple. My actual target comes from
’computed-file’.

-bjc

Re: Viewing derivation output in the store

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1773 bytes --]

Brian Cully schreef op wo 20-04-2022 om 11:40 [-0400]:
> 	How do I go from the derivation in ’test-drv’ to the output in
> the store? Something like how, from the REPL, this works:
> 
> ---[snip]---
> scheme@(guile-user)> ,enter-store-monad
> store-monad@(guile-user) [1]> (text-file "foo" "hello")
> $34 = "/gnu/store/mnzh1q6ilbw0bg04dg0vc4f47laz57lg-foo"
> ---[snip]---

Proposal, for 'plain-file', which returns a ‘high-level’ object that
can be lowered to a monadic value:

  ,run-in-store (lower-object (plain-file "foo" "bar"))

for 'gexp->derivation', which directly returns a monadic value:

  ,run-in-store (gexp->derivation "x" #~(pk #$output))

On my x86_64 system I can low look at the derivation at
‘/gnu/store/2s5gfk7a41v7ianfw5qv88y6336mvkw4-x.drv’.  However, the
derivation hasn't been build yet, so let's lower it.

To actually make the output path exist, the derivation needs to be
actually built:

;; XXX: store-lift and this construct is undocumented
,run-in-store (mlet* %store-monad ((drv (gexp->derivation "x" #~(pk #$output)))) ((store-lift build-derivations) (list drv)))
building path(s) `/gnu/store/kgidgrxjhg8w3hy3fhzjngwbix40nm8v-x'

;;; ("/gnu/store/kgidgrxjhg8w3hy3fhzjngwbix40nm8v-x")
builder for `/gnu/store/2s5gfk7a41v7ianfw5qv88y6336mvkw4-x.drv' failed
to produce output path `/gnu/store/kgidgrxjhg8w3hy3fhzjngwbix40nm8v-x'
While executing meta-command:
ERROR:
  1. &store-protocol-error:
      message: "build of `/gnu/store/2s5gfk7a41v7ianfw5qv88y6336mvkw4-x.drv' failed"
      status: 1

Maybe there's an easier way to do it -- maybe there could be a monadic
lower-to-store-item (like lower-object, but lowers to the store item
and not just the derivation).

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Viewing derivation output in the store

[Josselin Poiret]

Hello everyone,

Just for completeness' sake, here's my take on it from yesterday while
we were talking about it with Brian:

--8<---------------cut here---------------start------------->8---
(define-module (test)
  #:use-module (guix gexp)
  #:use-module (guix monads)
  #:use-module (guix derivations)
  #:use-module (guix store))

(define test-gexp
  #~(begin
      (copy-file #$(plain-file "helloes.txt" "contents") #$output)
      (format #t "Helloes~%")))

(with-store store
  (run-with-store store
    (mlet* %store-monad
        ((drv (gexp->derivation "myderivation" test-gexp))
         (output -> (derivation->output-path drv)))
      (mbegin %store-monad
        (built-derivations (list drv))
        (return (format #t "~a~%" output))))))
--8<---------------cut here---------------end--------------->8---

I tried to use the most monadic approach here to demonstrate its
usefulness, however, it seems to me that derivation->output-path is not
documented, along with built-derivations (which is just (store-lift
build-derivations)).

Lesson learned: don't shadow gexp (hence `test-gexp`)!

As an aside: is there anything preventing us from having do notation à
la Haskell?  This could help bridge mlet and mbegin with >>=, which in
its current form is impractical.  Here's what it could look like:

--8<---------------cut here---------------start------------->8---
(mdo %store-monad
  (drv <- (gexp-derivation "myderivation" test-gexp))
  (output <- (return (derivation->output-path drv)))
  (built-derivations (list drv))
  (return (format #t "~a~%" output)))
--8<---------------cut here---------------end--------------->8---

We could even have some more sugar for (x <- (return y)), Haskell has
`let x = y`, but we could instead have something like `(x := y)`?

If it's okay I could whip up a quick implementation for it, shouldn't be
too hard.

Best,
-- 
Josselin Poiret

Re: Viewing derivation output in the store

[Brian Cully]

Josselin Poiret <dev@jpoiret.xyz> writes:

> Just for completeness' sake, here's my take on it from yesterday while
> we were talking about it with Brian:

	Thank you for that, it was very enlightening.

> (define-module (test)
>   #:use-module (guix gexp)
>   #:use-module (guix monads)
>   #:use-module (guix derivations)
>   #:use-module (guix store))
>
> (define test-gexp
>   #~(begin
>       (copy-file #$(plain-file "helloes.txt" "contents") #$output)
>       (format #t "Helloes~%")))
>
> (with-store store
>   (run-with-store store
>     (mlet* %store-monad
>         ((drv (gexp->derivation "myderivation" test-gexp))
>          (output -> (derivation->output-path drv)))
>       (mbegin %store-monad
>         (built-derivations (list drv))
>         (return (format #t "~a~%" output))))))

	I tried playing with this for a while and ran into an issue when
actually executing it in a REPL: if the derivation hasn’t yet been
built, then during execution of of the ‘built-derivations’, an error is
thrown:
---[snip]---
ice-9/boot-9.scm:1685:16: In procedure raise-exception:
In procedure display: Wrong type argument in position 2: #<closed: string 7fb2850f3bd0>

Entering a new prompt.  Type `,bt' for a backtrace or `,q' to continue.
scheme@(zgrub) [1]> ,bt
In ice-9/boot-9.scm:
  1752:10  9 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
In /home/bjc/guix/guix/store.scm:
   658:37  8 (thunk)
  2129:24  7 (run-with-store #<store-connection 256.99 7fb2707783c0> _ #:guile-for-build _ #:system _ #:target _)
In current input:
   6373:8  6 (_ _)
In /home/bjc/guix/guix/store.scm:
  2001:38  5 (_ #<store-connection 256.99 7fb2748609b0>)
  1421:15  4 (_ #<store-connection 256.99 7fb2748609b0> _ _)
   759:13  3 (process-stderr _ _)
In unknown file:
           2 (display "building path(s) `/gnu/store/5jdlbc8vf5d0hf4inzj9k7gq6r9k6xsy-grub.cfg'\n" #<closed: string 7fb2850f3bd0>)
In ice-9/boot-9.scm:
  1685:16  1 (raise-exception _ #:continuable? _)
  1685:16  0 (raise-exception _ #:continuable? _)
---[snip]---

	I’m way too new to all this to say for sure, but it seems to me
that the build log output is being sent to a string, which, at least in
the REPL, is closed prematurely (if it’s ever even opened).

	If I wait a bit for the derivation to build, executing the above
code works fine and produces output in the output path.

	Is this something that’s worthy of a bug report, or am I doing
things I’m not supposed to be doing by building derivations in a REPL?

> I tried to use the most monadic approach here to demonstrate its
> usefulness, however, it seems to me that derivation->output-path is not
> documented, along with built-derivations (which is just (store-lift
> build-derivations)).

	Yes, thankfully the code was either easy to infer from the name
(derivation->output-path) or easy to read (built-derivations).

> Lesson learned: don't shadow gexp (hence `test-gexp`)!
>
> As an aside: is there anything preventing us from having do notation à
> la Haskell?  This could help bridge mlet and mbegin with >>=, which in
> its current form is impractical.  Here's what it could look like:

	Because of the above error, and general cluelessness, I tried
removing the ‘mbegin’ and found that the error went away, but the
derivation wasn’t built. My (wrong) assumption was that the ‘mbegin’ was
redundant with the ‘mlet’ since I figured the body of the mlet was in
the monad context. The documentation also describes mbegin as akin to
mlet, although at least the argument list implies a difference with
careful reading: mlet has ‘body’ while mbegin uses ‘mexp’.

> (mdo %store-monad
>   (drv <- (gexp-derivation "myderivation" test-gexp))
>   (output <- (return (derivation->output-path drv)))
>   (built-derivations (list drv))
>   (return (format #t "~a~%" output)))
>
> We could even have some more sugar for (x <- (return y)), Haskell has
> `let x = y`, but we could instead have something like `(x := y)`?

	This would definitely be more intelligible to me, at least, and
I know precious little Haskell.

	One final note that’s worth covering: while I was digging
through code I noticed that file-like objects have an embedded gexp
derivation. In the documentation they are described as the declarative
counterpart of ‘gexp->derivation’, but since there is no documented way
of turning them directly into a derivation, I was at a loss for what to
do with them until someone in IRC suggested using the #~(copy-file …)
gexp above.

	It turns out that there are ‘gexp-compiler’ instructions for
these things, which will turn them back into derivations. However, there
doesn’t seem to be a way to call them directly, and the only way I could
find to invoke them was by using ‘lower-object’ (again found by
grep-find through the codebase). This makes an intuitive sense, but the
naming implies I’m missing some larger architectural picture here where
‘raising’ and ‘lowering’ are more precisely defined.

	So the copy-file gexp can be substituted by ‘(lower-object
test-gexp)’:
---[snip]---
(mlet* %store-monad ((drv (lower-object (plain-file …)))) …)
---[snip]---

	I don’t know how useful these observations are, but hopefully at
least a little. I know I’m not alone in trying to understand how to make
the leap from file-likes to actual files; it seems to come up on IRC
with some regularity. At a minimum, this email will get indexed by
search engines and hopefully answer some future questions from there.

-bjc

Re: Viewing derivation output in the store

[Ludovic Courtès]

Hi!

Josselin Poiret <dev@jpoiret.xyz> skribis:

> As an aside: is there anything preventing us from having do notation à
> la Haskell?  This could help bridge mlet and mbegin with >>=, which in
> its current form is impractical.  Here's what it could look like:
>
> (mdo %store-monad
>   (drv <- (gexp-derivation "myderivation" test-gexp))
>   (output <- (return (derivation->output-path drv)))
>   (built-derivations (list drv))
>   (return (format #t "~a~%" output)))

I guess we could do that, though it seems to me like a marginal
improvement over ‘mbegin’ and ‘mlet’.

Ludo’.