Discoverability at the REPL level

Discoverability at the REPL level

[zimoun]

Dear,

Preparing the online Guix Days, maybe this discussion is worth.  It
echoes first with the talks “Guix compared to Nix” then with the recent
discussion about Emacs-Guix [1].


The topic is discoverability at the REPL level.


Well, I have a proposal draft «“guix repl” and beyond» that I never
sent, where the ideas was to discuss ’~/.guile’ and how to extend Guix
ending with these questions:

 1. Does Guix want a system of aliases?  For example, let the script
 “~/.config/guix/scripts/foo.scm“ and then ‘guix foo’.

 2. How could the API be more discoverable?

 3. Is the experimental ‘guix repl --gui’ reasonable?


The #1 popup’ed up in #38529 [2,3] and it is not related to
discoverablity but not orthogonal either.

The #3 means open Guile-Studio or any other front-end and echoes the
recent discussion about GUI front-end [4].


Therefore, here materials about the point #2. :-)


(The attentive reader is waiting for parametrized package PoC :-) and a
first discussion and arguments are this long thread [5].)


Feed back and ideas welcome.  Especially about:

        There are probably several ways to address it, including the
        unbound-variable hints and documentation.


All the best,
simon


1: https://yhetil.org/guix-devel/87tuttci4z.fsf_-_@gnu.org
2: https://yhetil.org/guix-bugs/87y2jie1aj.fsf@gmail.com/
3: http://issues.guix.gnu.org/issue/38529#60
4: https://yhetil.org/guix-devel/CAF-xJgsynM3KSzuM__f9dSPUC0epJ2QKdFwDftiLhTTuMfaTxw@mail.gmail.com
5: https://yhetil.org/guix-devel/87woitz1xx.fsf@gnu.org/

-------------------- Start of forwarded message --------------------
From: zimoun <zimon.toutoune@gmail.com>
Cc: guix-days@gnu.org
Date: Fri, 13 Nov 2020 19:20:21 +0100

Hi Ludo,

On Mon, 09 Nov 2020 at 22:55, Ludovic Courtès <ludo@gnu.org> wrote:

> Yeah I was also unsure; my idea was to give possible directions I had in
> mind (hence the plural: “the ways forward”), with the understanding that
> it’s nothing but a personal choice.

At «apero» break today, I have just watched [1] (speed 1.5) about Nix
modules (I do not know if it makes sense in the Guix context, anyway).
From my opinion, one interesting point is discoverability.

Guile is still a bit mysterious to me but a lot of tooling could be done
in this area.  For example, I am directly grepping in the Guix sources
to find the function names or which module provides which function, etc.
When you come from Python or Haskell (to name the few I know), the
experience is poor.  Even if Emacs+Geiser+Completion helps.

For a concrete example of annoyance,

--8<---------------cut here---------------start------------->8---
$ guix repl
scheme@(guix-user)> ,a fold-packages
scheme@(guix-user)> ,use(gnu packages)
scheme@(guix-user)> ,a fold-packages
(gnu packages): fold-packages	#<procedure fold-packages (proc init #:optional modules #:key select?)>
scheme@(guix-user)> ,d fold-packages
Call (PROC PACKAGE RESULT) for each available package defined in one of
MODULES that matches SELECT?, using INIT as the initial value of RESULT.  It
is guaranteed to never traverse the same package twice.
--8<---------------cut here---------------end--------------->8---

So I have to know beforehand the module to import (and I personally do
not know how to list all the functions that the module exports) then use
,apropos and ,describe to know the signature and if it is what I want.

In an ideal world, the first ’,a’ could provide hint for the module to
’,use’ and the ’,d’ should provide both signature and docstring. Since
the naming is more or less regular ’,a package bag’ should list all the
functions containing (name or docstring) the both terms so it becomes
easy to find which module provide ’package->bag’.  Instead on relying to
ag+Ivy. ;-)


Well, just sharing random ideas that popped up watching [1]. :-)

1: <https://youtu.be/7sQa04olUA0?t=5183>


Cheers,
simon
-------------------- End of forwarded message --------------------

-------------------- Start of forwarded message --------------------
From: Ludovic Courtès <ludo@gnu.org>
Cc: guix-days@gnu.org
Date: Sun, 15 Nov 2020 12:25:29 +0100

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> At «apero» break today, I have just watched [1] (speed 1.5) about Nix
> modules (I do not know if it makes sense in the Guix context, anyway).
> From my opinion, one interesting point is discoverability.

I think some of the criticism that applies to Nix does not apply to
Guix: we have ‘guix search’ and ‘guix system search’, we have package
transformation options (which achieve some of what people want to do
with Nixpkgs overlays), source code location shows up in search results,
etc.

Representing packages as functions is Nixpkgs is the root of
non-discoverability.  Conversely, packages, service types, etc. are
top-level declarations.

And… we have Guile modules when it comes to structuring code.  :-)

What we’re seeing here is an example of a DSL that grows and grows until
it includes all the features that general-purpose languages have.

What we don’t have yet is parameterized packages, but that motivates me
to come up with a PoC.

> Guile is still a bit mysterious to me but a lot of tooling could be done
> in this area.  For example, I am directly grepping in the Guix sources
> to find the function names or which module provides which function, etc.
> When you come from Python or Haskell (to name the few I know), the
> experience is poor.  Even if Emacs+Geiser+Completion helps.
>
> For a concrete example of annoyance,
>
> $ guix repl
> scheme@(guix-user)> ,a fold-packages
> scheme@(guix-user)> ,use(gnu packages)
> scheme@(guix-user)> ,a fold-packages
> (gnu packages): fold-packages	#<procedure fold-packages (proc init #:optional modules #:key select?)>
> scheme@(guix-user)> ,d fold-packages
> Call (PROC PACKAGE RESULT) for each available package defined in one of
> MODULES that matches SELECT?, using INIT as the initial value of RESULT.  It
> is guaranteed to never traverse the same package twice.
>
> So I have to know beforehand the module to import (and I personally do
> not know how to list all the functions that the module exports) then use
> ,apropos and ,describe to know the signature and if it is what I want.
>
> In an ideal world, the first ’,a’ could provide hint for the module to
> ’,use’ and the ’,d’ should provide both signature and docstring. Since
> the naming is more or less regular ’,a package bag’ should list all the
> functions containing (name or docstring) the both terms so it becomes
> easy to find which module provide ’package->bag’.  Instead on relying to
> ag+Ivy. ;-)

Yes, but you’re talking about API discovery here, which is different
from discoverability for package options from the CLI.

There are probably several ways to address it, including the
unbound-variable hints and documentation.

Perhaps worth discussing on guix-devel!  (You can share this message if
you want.)

Ludo’.
-------------------- End of forwarded message --------------------

Re: Discoverability at the REPL level

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 2051 bytes --]

Hi zimoun,

On Sun, 15 Nov 2020 14:02:04 +0100
zimoun <zimon.toutoune@gmail.com> wrote:

> In an ideal world, the first ’,a’ could provide hint for the module to
> ’,use’

There is no "the" module.  Any number of modules could have your searched-for
symbol--and the procedures so found could do completely unrelated things.  One
of the points of using modules in the first place is in order to group together
related things (and in order not to have unrelated things together).

What would be nice is for the module names to be easy to understand so you know
which module to import.  That's currently not great.  For example I have no
idea when something goes into (guix packages) vs (gnu packages).

Also, it would be nice and easy to implement to actually have the Guile REPL
search for all possible loadable modules that contain some symbol if it
encounters an unknown symbol, and print those, too (Guix often already does
that anyway!).

It should be easy to add such a thing to the guile repl.  In addition to
",describe" and ",apropos" there would be ",search" which would loop through
all modules, find the specified symbol and then print the docstrings of each
of those, including the module to use for each.

But since these modules can contain code that runs at module import time,
that's maybe also not what you want to actually happen (it would execute
code of random modules that are in the search path).
Then again, guile has declarative modules, too.  If those don't do that,
maybe just search in those.

Also, maybe you don't want Guile to actually IMPORT things into your namespace
when you do ",search".  You just want guile to list them.  That would be the
only complication.

... in before someone points out that this already exists in guile ;)

> and the ’,d’ should provide both signature and docstring.

I agree.  It's useless to separate those.  I mean it's nice that ",a"
exists--but then having ",d" not list the signature is just causing more work
for the developer.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Discoverability at the REPL level

[Ludovic Courtès]

Hi!

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> On Sun, 15 Nov 2020 14:02:04 +0100
> zimoun <zimon.toutoune@gmail.com> wrote:
>
>> In an ideal world, the first ’,a’ could provide hint for the module to
>> ’,use’
>
> There is no "the" module.  Any number of modules could have your searched-for
> symbol--and the procedures so found could do completely unrelated things.  One
> of the points of using modules in the first place is in order to group together
> related things (and in order not to have unrelated things together).

Yup.

> What would be nice is for the module names to be easy to understand so you know
> which module to import.  That's currently not great.  For example I have no
> idea when something goes into (guix packages) vs (gnu packages).

There’s a rationale: (guix …) is for Guix-the-tool (the mechanisms)
whereas (gnu …) is for the distribution.  Thus,
‘specification->package’, which browses (gnu packages …), is in (gnu
packages).

Module names are chosen to reflect the architecture of the code, so it
can be opaque to a newcomer.  I’m sure we can improve but there’s
necessarily that limitation.

> Also, it would be nice and easy to implement to actually have the Guile REPL
> search for all possible loadable modules that contain some symbol if it
> encounters an unknown symbol, and print those, too (Guix often already does
> that anyway!).
>
> It should be easy to add such a thing to the guile repl.  In addition to
> ",describe" and ",apropos" there would be ",search" which would loop through
> all modules, find the specified symbol and then print the docstrings of each
> of those, including the module to use for each.
>
> But since these modules can contain code that runs at module import time,
> that's maybe also not what you want to actually happen (it would execute
> code of random modules that are in the search path).
> Then again, guile has declarative modules, too.  If those don't do that,
> maybe just search in those.
>
> Also, maybe you don't want Guile to actually IMPORT things into your namespace
> when you do ",search".  You just want guile to list them.  That would be the
> only complication.

Yeah the unbound-variable hint, for example, currently looks at
already-loaded modules.  Triggering extra loads could have undesirable
side effects: I/O storm, increased memory usage, unwanted code executed,
etc.

Likewise, bindings, docstrings, etc. are all things that exist in a live
Guile system.  So searching them normally involves loading all the code,
which is unreasonable.

Now, .go files are ELF these days, and they contain docstrings and a
symbol table.  So one could implement a module search that parses ELF
files, browses docstrings and symbols, thus without ever running code.
Andy, if you read this, what are your thoughts?

Thanks,
Ludo’.

Re: Discoverability at the REPL level

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 935 bytes --]

zimoun 写道：
>  1. Does Guix want a system of aliases?  For example, let the 
>  script
>  “~/.config/guix/scripts/foo.scm“ and then ‘guix foo’.

I asked Guix and it said no, that's madness, sorry.  I think I 
agree.

Letting packages like Julien's excellent guix-home-manager[0] 
extend guix is interesting (once it works[1] :-).

It feels natural enough and allows anyone on a ‘guix foo is 
broken’ support call to ‘guix install guix-foo && guix foo’ with 
little trouble.

Guix executing arbitrary scripts in ~/.config does neither.

I'm unconvinced there's a use case that justifies reinventing 
shell aliases in Guix, in a heavy-weight fashion, while 
simultaneously not being ‘worth’ a real Guixtension package.

Kind regards,

T G-R

[0]: https://framagit.org/tyreunom/guix-home-manager
[1]: 
https://framagit.org/tyreunom/guix-home-manager#creating-the-first-home-generation

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: Discoverability at the REPL level

[Bengt Richter]

Hi Simon,

On +2020-11-15 14:02:04 +0100, zimoun wrote:
> Dear,
> 
> Preparing the online Guix Days, maybe this discussion is worth.  It
> echoes first with the talks “Guix compared to Nix” then with the recent
> discussion about Emacs-Guix [1].
> 
> 
> The topic is discoverability at the REPL level.
> 
> 
> Well, I have a proposal draft «“guix repl” and beyond» that I never
> sent, where the ideas was to discuss ’~/.guile’ and how to extend Guix
> ending with these questions:
> 
>  1. Does Guix want a system of aliases?  For example, let the script
>  “~/.config/guix/scripts/foo.scm“ and then ‘guix foo’.
> 
>  2. How could the API be more discoverable?
>

I find this bash script useful:
--8<---------------cut here---------------start------------->8---
#!/usr/bin/bash
echo "----from apropos:"
guile <<EOF | grep ": ${1:-help}"
,a ${1:-help}
EOF
guile <<EOF | tail -n +9
(display "----describe:\n")
,d  ${1:-help}
(display "----\n")
EOF
--8<---------------cut here---------------end--------------->8---
Just name it, e.g., guap (for GU.ile AP.ropos :)
chmod 755 guap

then try e.g.
guap string=
--8<---------------cut here---------------start------------->8---
----from apropos:
(guile): string=	#<procedure string= (_ _ #:optional _ _ _ _)>
(guile): string=?	#<procedure string=? (#:optional _ _ . _)>
----describe:
- Scheme Procedure: string= s1 s2 [start1 [end1 [start2 [end2]]]]
     Return `#f' if S1 and S2 are not equal, a true value otherwise.
----
--8<---------------cut here---------------end--------------->8---

Have fun tailoring to suit yourself.
Maybe some variations on restricting
the apropos output better than my grep :)
Or add some other ,xxx stuff or guile code to taste.

Note that you can of course invoke guap from emacs
like inserting the output of any bash command, as I did
for the above snip (writing this in emacs as mutt's editor choice).

>  3. Is the experimental ‘guix repl --gui’ reasonable?
> 
> 
> The #1 popup’ed up in #38529 [2,3] and it is not related to
> discoverablity but not orthogonal either.
> 
> The #3 means open Guile-Studio or any other front-end and echoes the
> recent discussion about GUI front-end [4].
> 
> 
> Therefore, here materials about the point #2. :-)
> 
> 
> (The attentive reader is waiting for parametrized package PoC :-) and a
> first discussion and arguments are this long thread [5].)
> 
> 
> Feed back and ideas welcome.  Especially about:
> 
>         There are probably several ways to address it, including the
>         unbound-variable hints and documentation.
> 
> 
> All the best,
> simon
> 
> 
> 1: https://yhetil.org/guix-devel/87tuttci4z.fsf_-_@gnu.org
> 2: https://yhetil.org/guix-bugs/87y2jie1aj.fsf@gmail.com/
> 3: http://issues.guix.gnu.org/issue/38529#60
> 4: https://yhetil.org/guix-devel/CAF-xJgsynM3KSzuM__f9dSPUC0epJ2QKdFwDftiLhTTuMfaTxw@mail.gmail.com
> 5: https://yhetil.org/guix-devel/87woitz1xx.fsf@gnu.org/
> 
[...]
-- 
Regards,
Bengt Richter

Re: Discoverability at the REPL level

[zimoun]

Hi Tobias,

On Mon, 16 Nov 2020 at 15:03, Tobias Geerinckx-Rice <me@tobias.gr> wrote:

> I'm unconvinced there's a use case that justifies reinventing 
> shell aliases in Guix, in a heavy-weight fashion, while 
> simultaneously not being ‘worth’ a real Guixtension package.

For example, I have tried to implement something to deal with
Reproducibility issues [1].  For instance, I run:

   guix repl -L . -- weather-repro.scm | sort | grep ghc

to list (almost) all the unreproducible Haskell packages.  What I would
like (not implemented yet) is to be able to filter by build system for
example, i.e., adds options.  Concretely, let imagine:

   guix repl -L . \
        -- weather-repro.scm --sort=reverse --build-system=haskell,ocaml

which simply becomes:

   guix weather-repro --sort=reverse --build-system=haskell,ocaml

Shell aliases are too limited, if I am not missing the obvious.

Crazier example, use it inside the “time-machine”:

   guix time-machine --branch=version-1.2.0 \
        -- weather-repro --sort=reverse --build-system=haskell,ocaml

which is not affordable via shell aliases.


Aside the power it should provide to the user hand, it could also be
helpful to easily test variant CLI propositions, for example:

  guix env foo --inputs-of bar

doing what “guix environment bar --ad-hoc foo“ currently does.


Cheers,
simon

1: <https://yhetil.org/guix-devel/86mu0rt95k.fsf@gmail.com>