bug#56114: Guix does not have a documented general and practical procedure for lowering a single lowerable object to the /gnu/store/... string.

bug#56114: Guix does not have a documented general and practical procedure for lowering a single lowerable object to the /gnu/store/... string.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 2996 bytes --]

Variants of this appear repeatedly on #guix every now and then,
and also sometimes unasked by me not posted anywhere.

How do I run compile a package, computed-file or more generally a
lowerable object from the REPL?  Let's look in the manual:

> Note that the ‘(guix monad-repl)’ module extends the Guile REPL with
> new “meta-commands” to make it easier to deal with monadic
> procedures: ‘run-in-store’, and ‘enter-store-monad’.  The former is
> used to “run” a single monadic value through the store:
> 
>      scheme@(guile-user)> ,run-in-store (package->derivation hello)
>      $1 = #<derivation /gnu/store/...-hello-2.9.drv => ...>
> 

Um, I want the actual store item, not the derivation, and I'm looking
into lowering a non-package, let's look elsewhere:

> As it turns out, the call to ‘package->derivation’ can even be
> omitted since it will take place implicitly, as we will see later
> (*note G-Expressions::):
> 
>      (define (sh-symlink)
>        (gexp->derivation "sh"
>                          #~(symlink (string-append #$bash
> "/bin/bash")
>                                     #$output)))
> 
>    Calling the monadic ‘sh-symlink’ has no effect.  As someone once
> said, “you exit a monad like you exit a building on fire: by
> running”. So, to exit the monad and get the desired effect, one must
> use ‘run-with-store’:
> 
>      (run-with-store (open-connection) (sh-symlink))
>      ⇒ /gnu/store/...-sh-symlink

Let's try this (using the REPL command):

scheme@(guix-user)> ,run-in-store (plain-file "foo" "bar")
While executing meta-command:
Wrong type to apply: #<<plain-file> name: "foo" content: "bar"
references: ()>

doesn't work, let's try the 'lower-object' procedure which I vaguely
recall to helped me out in the past but doesn't have any actual
examples in the manual:

scheme@(guix-user)> ,run-in-store (lower-object (plain-file "foo" "bar"))
$15 = "/gnu/store/798sxvdgr0680czdggbls7rd3sfwk2yx-foo"

This looks useful, let's try it for mixed-text-file:

scheme@(guix-user)> ,run-in-store (lower-object (mixed-text-file "foo"
"bar"))
$17 = #<derivation /gnu/store/ah3yfw0nvw6jaif56r121sgwjiygmvrk-foo.drv
=> /gnu/store/raihpcxdz6wmdpyj67bwd4bjk02xq0x3-foo 7f862e53f820>

nooo there's the #<derivation ...> again how do I get a compiled mixed-
text-file from the REPL?

(I mean I could use 'build-derivations' from (guix)The Store, but that
accepts a list, not an individual derivation, and it doesn't accept strings
according to the documentation so it cannot be used for the 'plain-file'
and aren't derivations a super-low-level thing for which G-exps and such
were intended to make the low level thing almost an implementation detail?)

Seems like we are missing a general & documented & simple to use
'lower-object-completely' (or maybe 'compile-object'?) procedure for
this ...  And maybe also a ,build-object REPL command?

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

bug#56114: Guix does not have a documented general and practical procedure for lowering a single lowerable object to the /gnu/store/... string.

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1886 bytes --]

Hi Maxime,

Maxime Devos <maximedevos@telenet.be> skribis:

> Seems like we are missing a general & documented & simple to use
> 'lower-object-completely' (or maybe 'compile-object'?) procedure for
> this ...  And maybe also a ,build-object REPL command?

How about the attached patch?

Sample session:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,use(gnu packages base)
scheme@(guile-user)> ,build coreutils
$11 = "/gnu/store/y8933036ljrz4ah9zcph09nmvdmmv5sf-coreutils-8.32-debug"
$12 = "/gnu/store/8fpk2cja3f07xls48jfnpgrzrljpqivr-coreutils-8.32"
scheme@(guile-user)> ,lower coreutils
$13 = #<derivation /gnu/store/nc93q3hmlzcgdn6jr7dv3j2m50ivn55f-coreutils-8.32.drv => /gnu/store/y8933036ljrz4ah9zcph09nmvdmmv5sf-coreutils-8.32-debug /gnu/store/8fpk2cja3f07xls48jfnpgrzrljpqivr-coreutils-8.32 7f0ebaf821e0>
scheme@(guile-user)> ,build (computed-file "foo" #~(begin (display "hi!\n")(mkdir #$output)(mkdir #$output:lib)))
$14 = "/gnu/store/axij9bkg56xv3k1z0l20gd6b0swn14sy-foo-lib"
$15 = "/gnu/store/h5s7k7m0fxk9n7q7729l3n4q7vyxnvpy-foo"
scheme@(guile-user)> ,build (computed-file "foo" #~(begin (display "Goeden dag!\n")(mkdir #$output)))
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://guix.bordeaux.inria.fr'... 100.0%
building /gnu/store/gplhka9g0bwv8b640zavw1z65y9zrvag-foo.drv...
$16 = "/gnu/store/ynvga6gxwj68mmk8ddj3i9sjhavkqah1-foo"
--8<---------------cut here---------------end--------------->8---

“lower” does what you would expect; “build” yields one value per output.

If that works for you, I’ll update the manual accordingly, and we can
always add more features (like build options) later on.  Thoughts?

Thanks,
Ludo’.


[-- Attachment #2: Type: text/x-patch, Size: 3685 bytes --]

diff --git a/guix/monad-repl.scm b/guix/monad-repl.scm
index aefabdeebb..15c10efe01 100644
--- a/guix/monad-repl.scm
+++ b/guix/monad-repl.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2014, 2015, 2016 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2014, 2015, 2016, 2022 Ludovic Courtès <ludo@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -21,6 +21,12 @@ (define-module (guix monad-repl)
   #:use-module (guix monads)
   #:use-module (guix utils)
   #:use-module (guix packages)
+  #:use-module (guix status)
+  #:autoload   (guix gexp) (lower-object)
+  #:use-module ((guix derivations)
+                #:select (derivation?
+                          derivation->output-paths built-derivations))
+  #:use-module (ice-9 match)
   #:use-module (ice-9 pretty-print)
   #:use-module (system repl repl)
   #:use-module (system repl common)
@@ -69,16 +75,56 @@ (define (store-monad-language store)
                          #:guile-for-build guile)
                     'store-monad)))
 
+(define %build-verbosity 1)
+
+(define* (evaluate/print-with-store mvalue #:key build?)
+  "Run monadic value MVALUE in the store monad and print its value."
+  (with-store store
+    (set-build-options store
+                       #:print-build-trace #t
+                       #:print-extended-build-trace? #t
+                       #:multiplexed-build-output? #t)
+    (with-status-verbosity %build-verbosity
+      (let* ((guile  (or (%guile-for-build)
+                         (default-guile-derivation store)))
+             (values (run-with-store store
+                       (if build?
+                           (mlet %store-monad ((obj mvalue))
+                             (if (derivation? obj)
+                                 (mbegin %store-monad
+                                   (built-derivations (list obj))
+                                   (return
+                                    (match (derivation->output-paths obj)
+                                      (((_ . files) ...) files))))
+                                 (return (list obj))))
+                           (mlet %store-monad ((obj mvalue))
+                             (return (list obj))))
+                       #:guile-for-build guile)))
+        (for-each (lambda (value)
+                    (run-hook before-print-hook value)
+                    (pretty-print value))
+                  values)))))
+
 (define-meta-command ((run-in-store guix) repl (form))
   "run-in-store EXP
 Run EXP through the store monad."
-  (with-store store
-    (let* ((guile (or (%guile-for-build)
-                      (default-guile-derivation store)))
-           (value (run-with-store store (repl-eval repl form)
-                                  #:guile-for-build guile)))
-      (run-hook before-print-hook value)
-      (pretty-print value))))
+  (evaluate/print-with-store (repl-eval repl form)))
+
+(define-meta-command ((verbosity guix) repl (level))
+  "verbosity LEVEL
+Change build verbosity to LEVEL."
+  (set! %build-verbosity level))
+
+(define-meta-command ((lower guix) repl (form))
+  "lower OBJECT
+Lower OBJECT into a derivation and return it."
+  (evaluate/print-with-store (lower-object (repl-eval repl form))))
+
+(define-meta-command ((build guix) repl (form))
+  "build OBJECT
+Lower OBJECT and build it, returning its output file name(s)."
+  (evaluate/print-with-store (lower-object (repl-eval repl form))
+                             #:build? #t))
 
 (define-meta-command ((enter-store-monad guix) repl)
   "enter-store-monad

bug#56114: Guix does not have a documented general and practical procedure for lowering a single lowerable object to the /gnu/store/... string.

[zimoun]

Hi,

If I read correctly, this snippet from the manual,

>>      scheme@(guile-user)> ,run-in-store (package->derivation hello)
>>      $1 = #<derivation /gnu/store/...-hello-2.9.drv => ...>

becomes

    scheme@(guile-user)> ,lower hello

and this snippet,

> scheme@(guix-user)> ,run-in-store (lower-object (plain-file "foo" "bar"))
> $15 = "/gnu/store/798sxvdgr0680czdggbls7rd3sfwk2yx-foo"

becomes

    scheme@(guile-user)> ,build (plain-file "foo" "bar")


However, note that,

--8<---------------cut here---------------start------------->8---
scheme@(guix-user)> ,build (plain-file "foo" "bar")
$1 = "/gnu/store/798sxvdgr0680czdggbls7rd3sfwk2yx-foo"
scheme@(guix-user)> ,lower (plain-file "foo" "bar")
$2 = "/gnu/store/798sxvdgr0680czdggbls7rd3sfwk2yx-foo"
--8<---------------cut here---------------end--------------->8---

which is confusing, especially when,

--8<---------------cut here---------------start------------->8---
scheme@(guix-user)> ,lower (mixed-text-file "foo" "bar")
$6 = #<derivation /gnu/store/40982nzm6c7n4yawvs3m39k313jsrl80-foo.drv => /gnu/store/raihpcxdz6wmdpyj67bwd4bjk02xq0x3-foo 7f6f9bc5cd20>
scheme@(guix-user)> ,build (mixed-text-file "foo" "bar")
$7 = "/gnu/store/raihpcxdz6wmdpyj67bwd4bjk02xq0x3-foo"
--8<---------------cut here---------------end--------------->8---


The issue – unrelated to the nice patch – is a potential “inconsistency”
between ’<plain-file>’ and ’<computed-file>’; ’plain-file’ refers to
’text-file’ and ’mixed-text-file’ refers to ’text-file*’; and both are
really different despite close names around. :-)

Maybe the docstring of ’plain-file’ and of ’mixed-text-file’ could be
extended.  For instance,

--8<---------------cut here---------------start------------->8---
(define (plain-file name content)
  "Return an lowered object representing a text file called NAME with the given
CONTENT (a string) to be added to the store.

This is the declarative counterpart of 'text-file'."
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
(define* (mixed-text-file name #:key guile #:rest text)
  "Return an object representing store file NAME containing TEXT.  TEXT is a
sequence of strings and file-like objects, as in:

  (mixed-text-file \"profile\"
                   \"export PATH=\" coreutils \"/bin:\" grep \"/bin\")

This is the declarative counterpart of 'text-file*' but note that a
'computed-file' object is returned."
--8<---------------cut here---------------end--------------->8---



> diff --git a/guix/monad-repl.scm b/guix/monad-repl.scm
> index aefabdeebb..15c10efe01 100644
> --- a/guix/monad-repl.scm
> +++ b/guix/monad-repl.scm
> @@ -1,5 +1,5 @@

All LGTM.


> +(define* (evaluate/print-with-store mvalue #:key build?)
> +  "Run monadic value MVALUE in the store monad and print its value."

I do not know if it makes sense to have a list of monadic values,
allowing something like:

    ,build hello coreutils



Cheers,
simon

bug#56114: Guix does not have a documented general and practical procedure for lowering a single lowerable object to the /gnu/store/... string.

[Ludovic Courtès]

Hello!

zimoun <zimon.toutoune@gmail.com> skribis:

> However, note that,
>
> scheme@(guix-user)> ,build (plain-file "foo" "bar")
> $1 = "/gnu/store/798sxvdgr0680czdggbls7rd3sfwk2yx-foo"
> scheme@(guix-user)> ,lower (plain-file "foo" "bar")
> $2 = "/gnu/store/798sxvdgr0680czdggbls7rd3sfwk2yx-foo"
>
>
> which is confusing, especially when,
>
> scheme@(guix-user)> ,lower (mixed-text-file "foo" "bar")
> $6 = #<derivation /gnu/store/40982nzm6c7n4yawvs3m39k313jsrl80-foo.drv => /gnu/store/raihpcxdz6wmdpyj67bwd4bjk02xq0x3-foo 7f6f9bc5cd20>
> scheme@(guix-user)> ,build (mixed-text-file "foo" "bar")
> $7 = "/gnu/store/raihpcxdz6wmdpyj67bwd4bjk02xq0x3-foo"

Right, that’s because ‘plain-file’ lowers directly to a store item, not
to a derivation.

I explicitly made “,build” hide that difference; I thought one would
just want “,build” to do the right thing and display the resulting store
item regardless of that kind of detail.

Conversely, “,lower” is for die-hard hackers and it doesn’t hide any
subtlety.

>> +(define* (evaluate/print-with-store mvalue #:key build?)
>> +  "Run monadic value MVALUE in the store monad and print its value."
>
> I do not know if it makes sense to have a list of monadic values,
> allowing something like:
>
>     ,build hello coreutils

I don’t think the REPL can support this syntax, but we could add support
for:

  ,build (list hello coreutils)

Thanks for your feedback!

Ludo’.

bug#56114: Guix does not have a documented general and practical procedure for lowering a single lowerable object to the /gnu/store/... string.

[zimoun]

Hi,

On Mon, 04 Jul 2022 at 22:13, Ludovic Courtès <ludo@gnu.org> wrote:

> I explicitly made “,build” hide that difference; I thought one would
> just want “,build” to do the right thing and display the resulting store
> item regardless of that kind of detail.
>
> Conversely, “,lower” is for die-hard hackers and it doesn’t hide any
> subtlety.

Both make sense. :-)


> Right, that’s because ‘plain-file’ lowers directly to a store item, not
> to a derivation.

My comment was about an unrelated thing.  Well, I do not use every day
G-exp nor ’plain-file’ and the like.  The interface appears to me
unexpected each time I use it but hey I can live with it. :-) However,
the documentation (or docstring) does not appear clear, each time I have
my «aaah right!» moment.  For instance, the manual says:

        The local-file, plain-file, computed-file, program-file, and
        scheme-file procedures below return file-like objects. That is,
        when unquoted in a G-expression, these objects lead to a file in
        the store.

and the docstrings say:

--8<---------------cut here---------------start------------->8---
(define (plain-file name content)
  "Return an object representing a text file called NAME with the given
CONTENT (a string) to be added to the store.

This is the declarative counterpart of 'text-file'."
--8<---------------cut here---------------end--------------->8---

or

--8<---------------cut here---------------start------------->8---
(define* (computed-file name gexp
                        #:key guile (local-build? #t) (options '()))
  "Return an object representing the store item NAME, a file or directory
computed by GEXP.  When LOCAL-BUILD? is #t (the default), it ensures the
corresponding derivation is built locally.  OPTIONS may be used to pass
additional arguments to 'gexp->derivation'.

This is the declarative counterpart of 'gexp->derivation'."
--8<---------------cut here---------------end--------------->8---

when ’plain-file’ and ’computed-file’ does not return the same type of
“file-like object“.  Especially when reading,

--8<---------------cut here---------------start------------->8---
(define* (mixed-text-file name #:key guile #:rest text)
  "Return an object representing store file NAME containing TEXT.  TEXT is a
sequence of strings and file-like objects, as in:

  (mixed-text-file \"profile\"
                   \"export PATH=\" coreutils \"/bin:\" grep \"/bin\")

This is the declarative counterpart of 'text-file*'."
--8<---------------cut here---------------end--------------->8---

it is hard to guess beforehand that ’plain-file’ and ’mixed-text-file’
do not return the same type of file-like object.

Therefore, I was thinking to tweak these docstrings to better underline
their difference.  Well, maybe it is only me. :-)


Cheers,
simon


PS:
It remembers me unrelated [1] mentioning confusion about “file or
file-like object”. :-)

1: <https://yhetil.org/guix/CAKjmbcA+PNVCZRnp3oY+P4iAeu4pTvtwUMuVRYk_mFoJpO=jUQ@mail.gmail.com>

bug#56114: Guix does not have a documented general and practical procedure for lowering a single lowerable object to the /gnu/store/... string.

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> My comment was about an unrelated thing.

I don’t take comments about unrelated things.  ;-)

> Well, I do not use every day G-exp nor ’plain-file’ and the like.  The
> interface appears to me unexpected each time I use it but hey I can
> live with it. :-) However, the documentation (or docstring) does not
> appear clear, each time I have my «aaah right!» moment.

[...]

> it is hard to guess beforehand that ’plain-file’ and ’mixed-text-file’
> do not return the same type of file-like object.

As someone who writes gexps and manipulates “file-like objects”, it
doesn’t matter whether a thing lowers to a derivation or to a plain
store item.  What matters is that you can insert it in a gexp and it’ll
do the right thing.  That’s why the docstrings don’t insist on that
difference.

Hope it makes sense!

Ludo’.

bug#56114: Guix does not have a documented general and practical procedure for lowering a single lowerable object to the /gnu/store/... string.

[zimoun]

Hi,

On Tue, 5 Jul 2022 at 09:57, Ludovic Courtès <ludo@gnu.org> wrote:

> I don’t take comments about unrelated things.  ;-)

So one more. ;-)


> As someone who writes gexps and manipulates “file-like objects”, it
> doesn’t matter whether a thing lowers to a derivation or to a plain
> store item.  What matters is that you can insert it in a gexp and it’ll
> do the right thing.  That’s why the docstrings don’t insist on that
> difference.
>
> Hope it makes sense!

The difference is explained nowhere and to understand why the returned
call is different, then one has to read the code; it is what I did.
:-)

Well, "do the right thing" is an expectation based on an understanding
about what the "right thing" means. :-)  When I am manipulating
"file-like objects", I am always confused by what it means and thus it
appears to me that it does not "do the right thing" and it is hard to
understand why.

Yes it makes sense to not insistt on that difference once you have
clear understanding of this very difference.  However, when learning
G-Exp -- which is already unusual -- "file-like objects" and how to
manipulate them is hard to grasp because the difference is never
written down.  I still think the docstring or the manual should
mention such difference.

        Explicit is better than implicit.
        In the face of ambiguity, refuse the temptation to guess.

        – The Zen of Python, by Tim Peters –

Anyway!  That's unrelated. ;-)


Cheers,
simon

bug#56114: Guix does not have a documented general and practical procedure for lowering a single lowerable object to the /gnu/store/... string.

[jbranso--- via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 179 bytes --]

I was playing around with this today, and it is awesome!

I think it should be merged, because it has already helped me with my opensmtpd records project.

Thanks,

Joshua

[-- Attachment #2: Type: text/html, Size: 450 bytes --]

bug#56114: Guix does not have a documented general and practical procedure for lowering a single lowerable object to the /gnu/store/... string.

[Joshua Branson via Bug reports for GNU Guix]

Whoops.  I sent my original message to the list, not directly to ya'll.

,build and ,lower are awesome!  I reccomend that we add ludo's proposed
patch into guix.  It's awesome!

Thanks,

Joshua

bug#56114: Guix does not have a documented general and practical procedure for lowering a single lowerable object to the /gnu/store/... string.

[Ludovic Courtès]

Hi!

Thanks all for your support.  :-)

I’ve pushed these new REPL commands as
4ce7f1fb24a111f3e92d5b889d1271bebf109d09, including a new “Using Guix
Interactively” section in the manual to document it.  Feedback welcome!

I’m sure we can add more REPL commands as we better see what useful
shorthands would be welcome.

Cheers,
Ludo’.

bug#56114: Guix does not have a documented general and practical procedure for lowering a single lowerable object to the /gnu/store/... string.

[Maxime Devos]

Ludovic Courtès schreef op vr 15-07-2022 om 17:41 [+0200]:
> Hi!
> 
> Thanks all for your support.  :-)
> 
> I’ve pushed these new REPL commands as
> 4ce7f1fb24a111f3e92d5b889d1271bebf109d09, including a new “Using Guix
> Interactively” section in the manual to document it.  Feedback welcome!
> 
> I’m sure we can add more REPL commands as we better see what useful
> shorthands would be welcome.
> 
> Cheers,
> Ludo’.

Works nicely:

scheme@(guix-user)> ,build (@ (gnu packages base) hello)
[...]
$1 = "/gnu/store/s5pd3rnzymliafb4la5sca63j86xs0y0-hello-2.12.1"

though an actual documented _procedure_ (not command) seems to be
missing, so maybe the issue needs to be reopened later, don't know yet
...

Greetings,
Maxime.