Back from GHM

Back from GHM

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 602 bytes --]

Hello!

I don’t know how long it will be before videos are on-line, so in the
meantime the slides of my GHM talk are available at:

  http://git.savannah.gnu.org/cgit/guix/maintenance.git/plain/talks/ghm-2014/guix-ghm-2014.20140815.pdf

I made several demos, the last two of which were particularly
appreciated I think (guix.el and guix-web.)  So special thanks to David
and Alex.  :-)

There were about 50 people in the room, and I think the talk was rather
well received (if you attended, comments welcome :-)).

I’ll send an update when videos are available.

Thanks,
Ludo’.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: Back from GHM

[Amirouche Boubekki]

[-- Attachment #1: Type: text/plain, Size: 1237 bytes --]

Héllo,


2014-08-22 11:22 GMT+02:00 Ludovic Courtès <ludo@gnu.org>:

> Hello!
>
> I don’t know how long it will be before videos are on-line, so in the
> meantime the slides of my GHM talk are available at:
>
>
> http://git.savannah.gnu.org/cgit/guix/maintenance.git/plain/talks/ghm-2014/guix-ghm-2014.20140815.pdf
>
> I made several demos, the last two of which were particularly
> appreciated I think (guix.el and guix-web.)  So special thanks to David
> and Alex.  :-)
>
> There were about 50 people in the room, and I think the talk was rather
> well received (if you attended, comments welcome :-)).
>

Slides look very good. I think it was a good presentation of all the
concepts (I know of) behind guix and nix.

Domen Kožar did a presentation targeting Python people
<http://pyvideo.org/video/3036/rethinking-packaging-development-and-deployment>,
it might be of interest to people interested in Guix

The web app looks very good [https://gitorious.org/guix-web/guix-web/]

I'm happy to know that guix can know be installed
<http://www.gnu.org/software/guix/manual/html_node/System-Installation.html>
.


>
> I’ll send an update when videos are available.
>
> Thanks,
> Ludo’.
>

[-- Attachment #2: Type: text/html, Size: 2012 bytes --]

Re: Back from GHM

[Thompson, David]

On Mon, Aug 25, 2014 at 7:57 AM, Amirouche Boubekki
<amirouche.boubekki@gmail.com> wrote:
>
> Domen Kožar did a presentation targeting Python people, it might be of
> interest to people interested in Guix
>

That was very interesting, indeed.  Thanks for sharing.  I hope we can
make use of Nix's {pip,bower,node,...}2nix scripts and make Guix
versions.  But first, we need to write guix-shell!

> The web app looks very good [https://gitorious.org/guix-web/guix-web/]
>

Why thank you. :)

> I'm happy to know that guix can know be installed.
>

Give it a shot on a spare computer if you have one and let us know how it goes!

- Dave

Re: Back from GHM

[Amirouche Boubekki]

[-- Attachment #1: Type: text/plain, Size: 939 bytes --]

2014-08-25 15:47 GMT+02:00 Thompson, David <dthompson2@worcester.edu>:

> On Mon, Aug 25, 2014 at 7:57 AM, Amirouche Boubekki
> <amirouche.boubekki@gmail.com> wrote:
> >
> > Domen Kožar did a presentation targeting Python people, it might be of
> > interest to people interested in Guix
> >
>
> That was very interesting, indeed.  Thanks for sharing.  I hope we can
> make use of Nix's {pip,bower,node,...}2nix scripts and make Guix
> versions.



> But first, we need to write guix-shell!
>

What is guix-shell?


>
> > The web app looks very good [https://gitorious.org/guix-web/guix-web/]
> >
>
> Why thank you. :)
>
> > I'm happy to know that guix can know be installed.
> >
>
> Give it a shot on a spare computer if you have one and let us know how it
> goes!
>

I will try, I struggle a bit with my new laptop, it has libreboot on it.
Anyway, I'll try this week end for sure.


>
> - Dave
>

[-- Attachment #2: Type: text/html, Size: 1940 bytes --]

guix-shell?

[Ludovic Courtès]

"Thompson, David" <dthompson2@worcester.edu> skribis:

> That was very interesting, indeed.  Thanks for sharing.

Indeed, very cool.

> I hope we can make use of Nix's {pip,bower,node,...}2nix scripts and
> make Guix versions.  But first, we need to write guix-shell!

I think nix-shell is mostly equivalent to:

  eval `guix package --search-paths`

Although maybe it would be better to have a command that does it without
actually having to create a profile.

Thoughts?

Thanks,
Ludo’.

Re: guix-shell?

[David Thompson]

Ludovic Courtès <ludo@gnu.org> writes:

> "Thompson, David" <dthompson2@worcester.edu> skribis:
>
>> I hope we can make use of Nix's {pip,bower,node,...}2nix scripts and
>> make Guix versions.  But first, we need to write guix-shell!
>
> I think nix-shell is mostly equivalent to:
>
>   eval `guix package --search-paths`
>
> Although maybe it would be better to have a command that does it without
> actually having to create a profile.
>

Hmm, I thought that this was a perfect use-case for profiles.  I haven't
read the source to nix-shell yet, but I inferred that it essentially did
the following:

1) Create profile in project directory
2) Install dependencies
3) Run the desired shell
4) Prepend the new profile's /bin to $PATH
5) Set search path env vars

How would we do it without creating a profile?

-- 
David Thompson
Web Developer - Free Software Foundation - http://fsf.org
GPG Key: 0FF1D807
Support the FSF: https://fsf.org/donate

Re: guix-shell?

[David Thompson]

Ludovic Courtès <ludo@gnu.org> writes:

> "Thompson, David" <dthompson2@worcester.edu> skribis:
>
>> I hope we can make use of Nix's {pip,bower,node,...}2nix scripts and
>> make Guix versions.  But first, we need to write guix-shell!
>
> I think nix-shell is mostly equivalent to:
>
>   eval `guix package --search-paths`
>
> Although maybe it would be better to have a command that does it without
> actually having to create a profile.
>
> Thoughts?

Okay, so after digging into nix's source code a bit, I've learned a few
things.  nix-shell is really just a symlink to nix-build, and it doesn't
seem to use profiles at all!  When nix-build is called as 'nix-shell',
it's behavior changes.  From the manual: "The command nix-shell will
build the dependencies of the specified derivation, but not the
derivation itself."  I can see that the nix-build script exits before
adding the derivation to the store when called as 'nix-shell'.

Here's the relevant Perl source code:

    die "$0: a single derivation is required\n" if scalar @drvPaths != 1;
    my $drvPath = $drvPaths[0];
    $drvPath = (split '!',$drvPath)[0];
    $drvPath = readlink $drvPath or die "cannot read symlink ‘$drvPath’" if -l $drvPath;
    my $drv = derivationFromPath($drvPath);

    # Build or fetch all dependencies of the derivation.
    my @inputDrvs = grep { my $x = $_; (grep { $x =~ $_ } @envExclude) == 0 } @{$drv->{inputDrvs}};
    system("$Nix::Config::binDir/nix-store", "-r", "--no-output", "--no-gc-warning", @buildArgs, @inputDrvs, @{$drv->{inputSrcs}}) == 0
        or die "$0: failed to build all dependencies\n";

    # Set the environment.
    my $tmp = $ENV{"TMPDIR"} // $ENV{"XDG_RUNTIME_DIR"} // "/tmp";
    if ($pure) {
        foreach my $name (keys %ENV) {
            next if grep { $_ eq $name } ("HOME", "USER", "LOGNAME", "DISPLAY", "PATH", "TERM", "IN_NIX_SHELL", "TZ", "PAGER");
            delete $ENV{$name};
        }
        # NixOS hack: prevent /etc/bashrc from sourcing /etc/profile.
        $ENV{'__ETC_PROFILE_SOURCED'} = 1;
    }
    $ENV{'NIX_BUILD_TOP'} = $ENV{'TMPDIR'} = $ENV{'TEMPDIR'} = $ENV{'TMP'} = $ENV{'TEMP'} = $tmp;
    $ENV{'NIX_STORE'} = $Nix::Config::storeDir;
    $ENV{$_} = $drv->{env}->{$_} foreach keys %{$drv->{env}};

    # Run a shell using the derivation's environment.  For
    # convenience, source $stdenv/setup to setup additional
    # environment variables and shell functions.  Also don't lose
    # the current $PATH directories.
    my $rcfile = "$tmpDir/rc";
    writeFile(
        $rcfile,
        "rm -rf '$tmpDir'; " .
        'unset BASH_ENV; ' .
        '[ -n "$PS1" ] && [ -e ~/.bashrc ] && source ~/.bashrc; ' .
        ($pure ? '' : 'p=$PATH; ' ) .
        'dontAddDisableDepTrack=1; ' .
        '[ -e $stdenv/setup ] && source $stdenv/setup; ' .
        'if [ "$(type -t runHook)" = function ]; then runHook shellHook; fi; ' .
        ($pure ? '' : 'PATH=$PATH:$p; unset p; ') .
        'set +e; ' .
        '[ -n "$PS1" ] && PS1="\n\[\033[1;32m\][nix-shell:\w]$\[\033[0m\] "; ' .
        'unset NIX_ENFORCE_PURITY; ' .
        'unset NIX_INDENT_MAKE; ' .
        'shopt -u nullglob; ' .
        'unset TZ; ' . (defined $ENV{'TZ'} ? "export TZ='${ENV{'TZ'}}'; " : '') .
        $envCommand);
    $ENV{BASH_ENV} = $rcfile;
    exec($ENV{NIX_BUILD_SHELL} // "bash", "--rcfile", $rcfile);
    die;

(Thank you for choosing Scheme, btw)

I found it confusing to have the build script also handle the shell via
some conditional logic.  I think Guix could do it better.  I will
implement 'guix shell' if I can get your opinion on the best approach to
take.  Would you tweak guix/scripts/build.scm to handle the special case
or export the necessary procedures and create a guix/scripts/shell.scm
module?

Thanks!

-- 
David Thompson
Web Developer - Free Software Foundation - http://fsf.org
GPG Key: 0FF1D807
Support the FSF: https://fsf.org/donate

Re: guix-shell?

[Andreas Enge]

Hello,

could you maybe provide a specification of what guix-shell is supposed to do?
Not knowing nix-shell, I admit to being somewhat lost as to which problem
you are trying to solve.

Andreas

Re: guix-shell?

[David Thompson]

Andreas Enge <andreas@enge.fr> writes:

> Hello,
>
> could you maybe provide a specification of what guix-shell is supposed to do?
> Not knowing nix-shell, I admit to being somewhat lost as to which problem
> you are trying to solve.
>
> Andreas
>

I think the Nix manual explains it best:
http://nixos.org/nix/manual/#sec-nix-shell

Basically, it creates a shell environment in which some packages
specified in an input file are available.  A great use-case for this is
setting up development environments for software projects.  A
'shell.nix' file would live in the root of a project's source tree.  A
new developer would clone the repository, run 'nix-shell' and have an
environment that has everything needed to build and run the software.

Does that help?

-- 
David Thompson
Web Developer - Free Software Foundation - http://fsf.org
GPG Key: 0FF1D807
Support the FSF: https://fsf.org/donate

Re: guix-shell?

[Amirouche Boubekki]

[-- Attachment #1: Type: text/plain, Size: 3367 bytes --]

2014-08-26 22:31 GMT+02:00 David Thompson <dthompson2@worcester.edu>:

> Andreas Enge <andreas@enge.fr> writes:
>
> > Hello,
> >
> > could you maybe provide a specification of what guix-shell is supposed
> to do?
> > Not knowing nix-shell, I admit to being somewhat lost as to which problem
> > you are trying to solve.
> >
> > Andreas
> >
>
> I think the Nix manual explains it best:
> http://nixos.org/nix/manual/#sec-nix-shell
>
>

> Basically, it creates a shell environment in which some packages
> specified in an input file are available.




> A great use-case for this is setting up development environments for
> software projects.


The core idea behind this is to be able to have an easy way to develop
applications
without requiring a heavy VM. This is kind of a container but at the shell
level. It's more
lightweight than VM, or LXC or docker but doesn't help with security, it's
only for development.

You install libraries and else in this "shell environment" and they have a
higher priority
than user and root libraries, so they are used by the application your are
developing.

In Python at least, it is called virtualenv. It has been merged into python
dev branch. The thing
is that you can only reliably install pure python packages in the
virtualenv. Anything that requires
non python dependencies can be of trouble. For instance, I never used
virtualenv to use
another version of mysql... an example of non-pure python package that
gives trouble is "ipython".
If I recall correctly cython based dependencies are also troublesome.

That's said, it's very helpful. Most of the time you have no constraint or
bugs because of binary stuff.
You would have to be - multitasking a lot - to need to install two versions
of PostgreSQL in parallel.

One of the cons DevOps argue against the use of virtualenv is that it
bypass the rigourous
process of Debian stable & security *and* you have to manage two kinds of
dependencies which makes
devops work more difficult.

For developpers and DevOps another cons is that you must be very rigorous
with dependencies. Sometime
the dev will install a library globally, the shell env will require it and
find it in the global namespace and it will
use it. When time arrive to reproduce the dev/prod environement you
discover that a library is missing... with no
information about the version to use.


> A 'shell.nix' file would live in the root of a project's source tree.  A
> new developer would clone the repository, run 'nix-shell' and have an
> environment that has everything needed to build and run the software.
>

Another solution might to use docker containers [nix docker
<http://zef.me/6049/nix-docker#introducing-nix-docker>][recursive docker
<http://blog.docker.com/2013/09/docker-can-now-run-within-docker/>]. I'm
not sure what this solution will bring over nix-shell except that since
it's a container, no global library will be used by the application.
Otherwise said, you will need to
track dependeciens from the very beginning.

Docker is all the rage. VMware added support to it recently. This is seems
to be the preferred method to do all things Cloud cf. EC2 and Google
appengines clones (like heroku) for more data.


>
> Does that help?
>
> --
> David Thompson
> Web Developer - Free Software Foundation - http://fsf.org
> GPG Key: 0FF1D807
> Support the FSF: https://fsf.org/donate
>
>

[-- Attachment #2: Type: text/html, Size: 5235 bytes --]

Re: guix-shell?

[Ludovic Courtès]

David Thompson <dthompson2@worcester.edu> skribis:

> Hmm, I thought that this was a perfect use-case for profiles.  I haven't
> read the source to nix-shell yet, but I inferred that it essentially did
> the following:
>
> 1) Create profile in project directory
> 2) Install dependencies
> 3) Run the desired shell
> 4) Prepend the new profile's /bin to $PATH
> 5) Set search path env vars

Just to be clear, that’s something that can be done currently with:

  guix package -p foo -i one two three
  eval `guix package -p $PWD/foo --search-paths`
  export PATH="$PWD/foo/bin"

(We could easily get rid of the last step.)

Ludo’.

Re: guix-shell?

[Ludovic Courtès]

David Thompson <dthompson2@worcester.edu> skribis:

> Okay, so after digging into nix's source code a bit, I've learned a few
> things.  nix-shell is really just a symlink to nix-build, and it doesn't
> seem to use profiles at all!  When nix-build is called as 'nix-shell',
> it's behavior changes.  From the manual: "The command nix-shell will
> build the dependencies of the specified derivation, but not the
> derivation itself."

Ooh, OK.  Interesting.

> I can see that the nix-build script exits before adding the derivation
> to the store when called as 'nix-shell'.
>
> Here's the relevant Perl source code:

[...]

> (Thank you for choosing Scheme, btw)

Yup.  :-)

> I found it confusing to have the build script also handle the shell via
> some conditional logic.  I think Guix could do it better.

+1

> I will implement 'guix shell' if I can get your opinion on the best
> approach to take.  Would you tweak guix/scripts/build.scm to handle
> the special case or export the necessary procedures and create a
> guix/scripts/shell.scm module?

The problem with a command that spawns a shell IMO is that it does not
compose well: you get a new shell (which shell program is actually
run?), and you can influence a running shell, or use it in a script,
etc.

So instead I would imagine a command like:

  guix environment emacs

which would build the dependencies of Emacs, and then output the search
path as per ‘guix package --search-paths’, so that one can just source
it and be done.

In addition, one could do things like:

  guix environment -l foo.scm

Things like that.

WDYT?

Thanks,
Ludo’.

Re: guix-shell?

[David Thompson]

Ludovic Courtès <ludo@gnu.org> writes:

> David Thompson <dthompson2@worcester.edu> skribis:
>
>> I will implement 'guix shell' if I can get your opinion on the best
>> approach to take.  Would you tweak guix/scripts/build.scm to handle
>> the special case or export the necessary procedures and create a
>> guix/scripts/shell.scm module?
>
> The problem with a command that spawns a shell IMO is that it does not
> compose well: you get a new shell (which shell program is actually
> run?), and you can influence a running shell, or use it in a script,
> etc.
>
> So instead I would imagine a command like:
>
>   guix environment emacs
>
> which would build the dependencies of Emacs, and then output the search
> path as per ‘guix package --search-paths’, so that one can just source
> it and be done.
>
> In addition, one could do things like:
>
>   guix environment -l foo.scm

That sounds pretty good to me.  I don't have the process clear in my
head yet, though.  I need to read how our build script works more.

I think it would be nice to have a default file that guix environment
searches for if none is specified.  Tools like Ruby's bundler
automatically use a file called 'Gemfile'.  Perhaps guix environment
could look for an 'environment.scm' file by default?

-- 
David Thompson
Web Developer - Free Software Foundation - http://fsf.org
GPG Key: 0FF1D807
Support the FSF: https://fsf.org/donate

Re: guix-shell?

[Ludovic Courtès]

David Thompson <dthompson2@worcester.edu> skribis:

> I think it would be nice to have a default file that guix environment
> searches for if none is specified.  Tools like Ruby's bundler
> automatically use a file called 'Gemfile'.  Perhaps guix environment
> could look for an 'environment.scm' file by default?

Yes, why not.

Ludo’.

Re: guix-shell?

[David Thompson]

Ludovic Courtès <ludo@gnu.org> writes:

> David Thompson <dthompson2@worcester.edu> skribis:
>
>> I will implement 'guix shell' if I can get your opinion on the best
>> approach to take.  Would you tweak guix/scripts/build.scm to handle
>> the special case or export the necessary procedures and create a
>> guix/scripts/shell.scm module?
>
> The problem with a command that spawns a shell IMO is that it does not
> compose well: you get a new shell (which shell program is actually
> run?), and you can influence a running shell, or use it in a script,
> etc.
>
> So instead I would imagine a command like:
>
>   guix environment emacs
>
> which would build the dependencies of Emacs, and then output the search
> path as per ‘guix package --search-paths’, so that one can just source
> it and be done.
>
> In addition, one could do things like:
>
>   guix environment -l foo.scm
>

I started working on this.  As a test, I wrote a script that loads a
list of packages from a file, builds them, and then outputs the search
paths.  One crucial search path is missing though: PATH.  Should 'guix
environment' create a new profile with all of the necessary packages in
it?  IIRC, nix shell doesn't do this, but I don't fully understand why
and how.

-- 
David Thompson
Web Developer - Free Software Foundation - http://fsf.org
GPG Key: 0FF1D807
Support the FSF: https://fsf.org/donate

Re: guix-shell?

[Ludovic Courtès]

David Thompson <dthompson2@worcester.edu> skribis:

> I started working on this.  As a test, I wrote a script that loads a
> list of packages from a file, builds them, and then outputs the search
> paths.  One crucial search path is missing though: PATH.

The reason why it’s missing is that, search path specifications are
attached to packages, but there’s no obvious package to attach ‘PATH’
to.

However, it would be easy, and perhaps desirable, to change
‘search-path-environment-variables’ in package.scm to do that.

> Should 'guix environment' create a new profile with all of the
> necessary packages in it?  IIRC, nix shell doesn't do this, but I
> don't fully understand why and how.

The thing is, all the packages in question must be registered as GC
roots, at least for the duration of the shell session (otherwise they
could disappear.)

The easiest way to do that is to build a profile, and then register this
profile as a GC root (like what ‘guix build -r’ does.)

Another option would be to keep the connection to the daemon open (this
is possible if and only if ‘guix environment’ spawns the shell by
itself.)  Not so great, though.

HTH,
Ludo’.

Re: guix-shell?

[David Thompson]

Ludovic Courtès <ludo@gnu.org> writes:

> David Thompson <dthompson2@worcester.edu> skribis:
>
>> Should 'guix environment' create a new profile with all of the
>> necessary packages in it?  IIRC, nix shell doesn't do this, but I
>> don't fully understand why and how.
>
> The thing is, all the packages in question must be registered as GC
> roots, at least for the duration of the shell session (otherwise they
> could disappear.)
>

I abandoned the profile approach and tried to implement something
similar to what nix-shell does: Build all input derivations and set the
environment variables that a builder would set.  My primary reason for
using the low-level derivations instead of high-level package inputs is
that the derivations include information about all of the implicit
inputs as well.  However, building the correct environment has still
turned out to be very difficult, and I'm stuck once again.

My problems seem to stem from the disconnect between packages and
derivations, and the implicit inputs of build systems.  It's fairly easy
to construct PATH and standard-search-paths, but I haven't been able to
determine all of the other native search paths that a package might need
in a build system agnostic manner.  The derivation files do not store
such paths, they are in the source code of the builder script.
nix-shell has it easy because in Nix there is always a $stdenv/setup
file that can be sourced that does the right thing.

Since using derivations alone wasn't enough, I tried to use the
native-search-paths of the package inputs.  However, there is no
interface to obtain all of the inputs for a package, including the
implicit ones, in client-side code.  Without the implicit inputs, it
seems impossible to create an environment is the same as the environment
of a builder.

Do you have thoughts on how to overcome these issues?  I'm stumped.  I
could very well be overlooking something.  I hope the above paragraphs
have made some sense.

-- 
David Thompson
Web Developer - Free Software Foundation - http://fsf.org
GPG Key: 0FF1D807
Support the FSF: https://fsf.org/donate

Re: guix-shell?

[Ludovic Courtès]

David Thompson <dthompson2@worcester.edu> skribis:

> I abandoned the profile approach and tried to implement something
> similar to what nix-shell does: Build all input derivations and set the
> environment variables that a builder would set.  My primary reason for
> using the low-level derivations instead of high-level package inputs is
> that the derivations include information about all of the implicit
> inputs as well.

That makes sense.

What about:

  1. Computing the list of search-path-specifications of all the
     transitive inputs of the package, the way ‘package-derivation’
     does it;

  2. Computing, based on these specifications and all the input
     derivations, the actual search paths, the way ‘set-paths’ does it?

Unless I’m mistaken, this would be equivalent to what happens with
‘build-system’ implementations, so that should work.

HTH!
Ludo’.

Re: guix-shell?

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> What about:
>
>   1. Computing the list of search-path-specifications of all the
>      transitive inputs of the package, the way ‘package-derivation’
>      does it;
>
>   2. Computing, based on these specifications and all the input
>      derivations, the actual search paths, the way ‘set-paths’ does it?

As discussed on IRC, that won’t work because ‘build-system’
implementations augment the search path specifications on their own, and
there’s no good way from the outside to tell what these are.

Bummer.

Ludo’.