Scripting guix in guile

Scripting guix in guile

[Konrad Hinsen]

Hi everyone,

I'd like to write Guile scripts that inspect Guix, for example analyze
the packages is my profile. That turned out to be a lot more difficult
than I expected, and in fact I haven't found a satisfying general
solution yet, meaning a script that I could publish in such a way that
any Guix user could download and run it without modification.

The problems I see are

 1. What to put into the #! line to locate Guile.
 2. How to construct the load path to make sure it includes
    Guix as installed under $HOME/.config/guix/current

I wonder if could somehow hijack 'guix repl', which solves these
problems for a REPL but not for scripts.

Any ideas?

Konrad

Re: Scripting guix in guile

[Jesse Gibbons]

On Sat, 2019-08-31 at 19:03 +0200, Konrad Hinsen wrote:
> Hi everyone,
> 
> I'd like to write Guile scripts that inspect Guix, for example
> analyze
> the packages is my profile. That turned out to be a lot more
> difficult
> than I expected, and in fact I haven't found a satisfying general
> solution yet, meaning a script that I could publish in such a way
> that
> any Guix user could download and run it without modification.
> 
> The problems I see are
> 
>  1. What to put into the #! line to locate Guile.
I don't think you need a shebang to script guile. If you do, you could
always make a guix package for the scripts, and that will fix
everything in the patch-shebangs phase.
>  2. How to construct the load path to make sure it includes
>     Guix as installed under $HOME/.config/guix/current
I think the default GUILE_LOAD_PATH includes guix. Here's the contents
of mine:
~$ ls $GUILE_LOAD_PATH 
bytestructures  git.scm  gnutls      guix.scm  json.scm  shepherd.scm
gcrypt          gnu      gnutls.scm  ice-9/    mcron     sqlite3.scm
git             gnu.scm  guix        json      shepherd  ssh

gnu/, gnu.scm, guix/, and guix.scm are all guix, and guix is not part
of my current profile. Note also most of these are prerequisites for
guix.
> 
> I wonder if could somehow hijack 'guix repl', which solves these
> problems for a REPL but not for scripts.
> 
> Any ideas?
> 
> Konrad
> 
-You're writing tools for a package manager... try packaging those
tools for the package manager. It will take care of them like it takes
care of everything else. 
--You can wrap the executables with the environment variables you need
so the executable scripts are callable from the default PATH. See what
guix does with guile, mcron, and shepherd, which are mostly guile
scripts with a little bit of native code.

Re: Scripting guix in guile

[Jesse Gibbons]

On Sat, 2019-08-31 at 11:38 -0600, Jesse Gibbons wrote:
> On Sat, 2019-08-31 at 19:03 +0200, Konrad Hinsen wrote:
> > Hi everyone,
> > 
> > I'd like to write Guile scripts that inspect Guix, for example
> > analyze
> > the packages is my profile. That turned out to be a lot more
> > difficult
> > than I expected, and in fact I haven't found a satisfying general
> > solution yet, meaning a script that I could publish in such a way
> > that
> > any Guix user could download and run it without modification.
> > 
> > The problems I see are
> > 
> >  1. What to put into the #! line to locate Guile.
> 
> I don't think you need a shebang to script guile.
Turns out you do.
>  If you do, you could always make a guix package for the scripts, and
> that will fix everything in the patch-shebangs phase.
If you package it for guix you can use it to develop your scripts as
well. 
> >  2. How to construct the load path to make sure it includes
> >     Guix as installed under $HOME/.config/guix/current
> 
> I think the default GUILE_LOAD_PATH includes guix. Here's the
> contents
> of mine:
> ~$ ls $GUILE_LOAD_PATH 
> bytestructures  git.scm  gnutls      guix.scm  json.scm  shepherd.scm
> gcrypt          gnu      gnutls.scm  ice-9/    mcron     sqlite3.scm
> git             gnu.scm  guix        json      shepherd  ssh
> 
> gnu/, gnu.scm, guix/, and guix.scm are all guix, and guix is not part
> of my current profile. Note also most of these are prerequisites for
> guix.
> > 
> > I wonder if could somehow hijack 'guix repl', which solves these
> > problems for a REPL but not for scripts.
> > 
> > Any ideas?
> > 
> > Konrad
> > 
> 
> -You're writing tools for a package manager... try packaging those
> tools for the package manager. It will take care of them like it
> takes
> care of everything else. 
> --You can wrap the executables with the environment variables you
> need
> so the executable scripts are callable from the default PATH. See
> what
> guix does with guile, mcron, and shepherd, which are mostly guile
> scripts with a little bit of native code.

Re: Scripting guix in guile

[Konrad Hinsen]

Hi Jesse,

First of all, thanks for your suggestions!

>>  1. What to put into the #! line to locate Guile.
> I don't think you need a shebang to script guile. If you do, you could
> always make a guix package for the scripts, and that will fix
> everything in the patch-shebangs phase.

Right, packaging solves the problem because Guix takes care of patching
my script. But that comes at a significant cost in terms of deployment
complexity. It's certainly the right way to distribute black-box tools,
but my scripts are mainly illustrations for working with Guix. I want to
use them in a tutorial, saying "there is no built-in Guix functionality
for this, but you can write your own scripts interfacing with
Guix. Here's an example, download it, play with it, modify it to fit
your needs."

> I think the default GUILE_LOAD_PATH includes guix. Here's the contents

Not on my system. I am using Guix on top of Ubuntu, maybe for Guix
System it works out of the box. I'll try with a virtual machine.

> --You can wrap the executables with the environment variables you need
> so the executable scripts are callable from the default PATH. See what
> guix does with guile, mcron, and shepherd, which are mostly guile
> scripts with a little bit of native code.

Looking at those examples is certainly a good plan - I'll do and report
back!

Konrad.

Re: Scripting guix in guile

[Jesse Gibbons]

On Sun, 2019-09-01 at 10:15 +0200, Konrad Hinsen wrote:
> Hi Jesse,
> 
> First of all, thanks for your suggestions!
> 
> > >  1. What to put into the #! line to locate Guile.
> > 
> > I don't think you need a shebang to script guile. If you do, you
> > could
> > always make a guix package for the scripts, and that will fix
> > everything in the patch-shebangs phase.
> 
> Right, packaging solves the problem because Guix takes care of
> patching
> my script. But that comes at a significant cost in terms of
> deployment
> complexity. It's certainly the right way to distribute black-box
> tools,
> but my scripts are mainly illustrations for working with Guix. I want
> to
> use them in a tutorial, saying "there is no built-in Guix
> functionality
> for this, but you can write your own scripts interfacing with
> Guix. Here's an example, download it, play with it, modify it to fit
> your needs."
I'm don't know if there's a way to run raw scripts by themselves in
guixsd without patching the shebangs. I currently have three solutions
to this:

1. Tell your audience to call "guile <scriptname>" when running the
script. This does not garuntee guix is in their GUILE_LOAD_PATH though, so you will either need to tell them to add a segment to .bashrc to add the guix source location to their GUILE_LOAD_PATH or have them use the -L flag to have guile search that directory.

2. Instruct the audience to use features of guix when playing with the
script:
- guix build --source <packagename> can download the source into a
tarball (if it is a tarball) or a directory (if it is in a git repo).
If it is successful, it will output where the source is to stdout. If
the source is in a directory, you will need to copy it and chown the
copy before you can edit it.
- guix environment <packagename> will put everything the package needs
into a shell environment.
- guix pack <packagename> builds a tarball of the package with its
dependencies. If it is successful it will output where the tarball is
in the store to stdout.
- the --root= option of guix build --source and guix pack will make a soft link to the resulting tarball or directory in the specified location. - the --with-source=PACKAGE=SOURCE option can specify where the source is, so if you have local source you can specify its root directory.

3. I think you can use autogen to find out where guile and guix are
located and use that location in the shebang and in a script that adds
guix to GUILE_LOAD_PATH. This is a bit complicated though, and you will
need to make sure your audience has autogen before they can do
anything.
> 
> > I think the default GUILE_LOAD_PATH includes guix. Here's the
> > contents
> 
> Not on my system. I am using Guix on top of Ubuntu, maybe for Guix
> System it works out of the box. I'll try with a virtual machine.
> 
If you package your scripts, you can specify guix as an input. I think
that will either put guix in the GUILE_LOAD_PATH or instruct whoever
installs it to add a snippet to .bashrc that puts guix in
GUILE_LOAD_PATH.
> > --You can wrap the executables with the environment variables you
> > need
> > so the executable scripts are callable from the default PATH. See
> > what
> > guix does with guile, mcron, and shepherd, which are mostly guile
> > scripts with a little bit of native code.
> 
> Looking at those examples is certainly a good plan - I'll do and
> report
> back!
> 
> Konrad.

Re: Scripting guix in guile

[Ludovic Courtès]

Hi Konrad,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

> The problems I see are
>
>  1. What to put into the #! line to locate Guile.

On Guix System #!/run/current-system/profile/bin/guile is very likely to
work :-), but on other systems you could use “#!/usr/bin/env guile” I
suppose.

>  2. How to construct the load path to make sure it includes
>     Guix as installed under $HOME/.config/guix/current

That’s indeed annoying.  I think we should add ‘-s’ and similar flags
that ‘guile’ supports to ‘guix repl’.

In the meantime, as Jesse suggested, you could arrange to have both
‘guix’ (the package) and ‘guile’ in a profile, such that Guix and its
dependencies are available in $GUILE_LOAD_PATH.

As for inspecting the installed packages, the entry point is (guix
profiles), which has everything you need to grab info about a profile.

HTH!

Ludo’.

Re: Scripting guix in guile

[Konrad Hinsen]

Hi Jesse and Ludo,

Jesse Gibbons <jgibbons2357@gmail.com> writes:

> 1. Tell your audience to call "guile <scriptname>" when running the
> script.

That's very reasonable, but...

> This does not garuntee guix is in their GUILE_LOAD_PATH though, so you
> will either need to tell them to add a segment to .bashrc to add the
> guix source location to their GUILE_LOAD_PATH or have them use the -L
> flag to have guile search that directory.

that sounds a bit complicated to explain, especially since they first
have to figure out the correct path segment to add, which depends on how
Guix was installed.

> 2. Instruct the audience to use features of guix when playing with the
> script:

Some of those are in fact my main topic (which will be part of a
tutorial on managing software environments for reproducible research).
But I want to keep technical details to a minimum for an introduction to
Guix.

> 3. I think you can use autogen to find out where guile and guix are
> located and use that location in the shebang and in a script that adds
> guix to GUILE_LOAD_PATH. This is a bit complicated though, and you will
> need to make sure your audience has autogen before they can do
> anything.

I don't even use autogen myself, so that's perhaps not the right
approach...


Ludovic Courtès <ludo@gnu.org> writes:

>>  1. What to put into the #! line to locate Guile.
>
> On Guix System #!/run/current-system/profile/bin/guile is very likely to
> work :-), but on other systems you could use “#!/usr/bin/env guile” I
> suppose.

True. But I'd prefer something that works on both!

>>  2. How to construct the load path to make sure it includes
>>     Guix as installed under $HOME/.config/guix/current
>
> That’s indeed annoying.  I think we should add ‘-s’ and similar flags
> that ‘guile’ supports to ‘guix repl’.

That sounds like a good idea, though I'd probably use "guix script" or
something similar rather than overload "repl".

> In the meantime, as Jesse suggested, you could arrange to have both
> ‘guix’ (the package) and ‘guile’ in a profile, such that Guix and its
> dependencies are available in $GUILE_LOAD_PATH.

That's what I thought would work, but it doesn't:

   $ which guix
   /home/hinsen/.guix-profile/guix
   $ which guile
   /home/hinsen/.guix-profile/guile

   $ echo $GUILE_LOAD_PATH
   /home/hinsen/.guix-profile/share/guile/site/2.2

   $ ls $GUILE_LOAD_PATH
   emacs-guix  emacs-guix.scm  gcrypt

That's with a Guix install under Ubuntu.

>´ As for inspecting the installed packages, the entry point is (guix
> profiles), which has everything you need to grab info about a profile.

Indeed, together with (guix packages) that's enough for exploring
a profile or environment, which is my main topic. My scripts
output stuff like (transitive-manifest-entries ...) or
(package-closure ...) in a readable format, to let users get a better
idea of what their code really depends on, and how.

Cheers,
  Konrad.