The future of 'guix environment'

The future of 'guix environment'

[Thompson, David]

Hi all, been awhile!

I wrote the initial version of 'guix environment' almost 2 years ago,
and have used it constantly since.  It was a great start, but from my
own experiences and feedback from others, I have identified a number
of areas that need improvement.  Fixing them will require breaking
changes, but I think it's important to break things sooner rather than
later because after 1.0 it will be a hard sell.

By far the most annoying thing is that leaving and coming back to a
project (say, after a reboot) may entail rebuilding your entire
environment if you have updated Guix in between.  What was supposed to
be a quick hack turns into a long Guix maintenance session,
potentially building things from source and dealing with build
failures.

As an avid Emacs user, I want to do everything in Emacs, but if I want
to use, say, M-x compile but in the context of my development
environment, I have to wrap the command in 'guix environment' and now
everything takes way longer because Guix wastes tons of time talking
to the daemon to build a profile that I know already exists.   This
goes hand in hand with the previous issue.  We need a cache.

Another issue is that 'guix environment' doesn't have any sane
defaults to speak of.  In the Ruby world, the 'guix environment'
equivalent is called Bundler.  When you want to work on someone's Ruby
project, you enter the source tree and run 'bundle'. No other
arguments.  It fetches everything you need based upon a configuration
file, which by convention is named 'Gemfile'.  There's no good reason
why 'guix environment' cannot be this simple to use.

The default behavior of 'guix environment foo' creating an environment
with the *dependencies* of the package 'foo' seemed like a good idea
at the time (it's what nix-shell did, after all), but most of the time
when I define throwaway environments on the command-line (as opposed
to a guix.scm file) I *always* use the --ad-hoc flag.  Using the
packages themselves rather than the dependencies should be the
default.

The final issue is more of a wishlist thing, but I think it is
something that will make 'guix environment' far and away the most
powerful development environment setup tool there is.  We should be
able to define more than just what packages to include in the
environment.  We should be able to define services, too.  A great
use-case for such a feature is a web application.  Think about how
convenient it would be if a newcomer to a project could run 'guix
environment' and have nginx and mysql automatically launched for them
with the proper configuration.  The closest tool to this today would
be Docker's 'compose' tool, which can manage many containers, each
running a single service.  Surely we can do better than that!

If you've followed along this far, great!  Now here's my list of
proposed changes:

1) Add a caching mechanism.  The environment profile should get built
once, and then a symlink to it should be created in $PWD and
registered as a GC root.  This will, of course, require re-using some
'guix package' code to delete the profile.  For the sake of the rest
of this post, let's say that a --cache flag does this, and a --update
flag forces a rebuilding of the cached profile.

2) Make "ad-hoc" the default.  Remove the --ad-hoc flag and replace it
with a --dependencies flag instead, reversing the current behavior.
'guix environment --dependencies guix' would create a guix dev
environment, for example.

3) Change how --load behaves.  Instead of evaluating a file and
expected a package object in return, instead expect a
<guix-environment> record.  This would provide a declarative way to
specify the complete environment: which packages to pull in, whether
to purify the environment (--pure), whether to run in a container
(--container), whether to give the container network access
(--network), etc.  Command line flags would take precedence over what
is specified in the config file.  --load may only appear *once* in the
command line args, whereas now it many appear N times.

4) Make 'guix environment' with no other args operate like 'guix
environment --cache --load=guix.scm'.  'guix.scm' is a placeholder
name for whatever we decide the conventional name for an environment
config should be.  Throwaway environments (such as 'guix environment
ruby -- irb') would not have caching turned on by default, because
that would quickly become a burden.

5) Add support for Shepherd services.  When an environment is created,
a new Shepherd process is launched to handle all the necessary
services.  For a web application this could be nginx, mysql, redis,
etc.  This feature can be implemented later, even post 1.0, provided
we agree upon using a <guix-environment> record type.

After all these changes, a Guix user should be able to jump into a new
project, run 'guix environment' and be ready to go.  When they update
Guix and want to refresh their environment, they would run 'guix
environment --update'.

Thoughts?

- Dave

Re: The future of 'guix environment'

[宋文武]

"Thompson, David" <dthompson2@worcester.edu> writes:

> Hi all, been awhile!

Yeah, glad to see you!


> [...]

>
> If you've followed along this far, great!  Now here's my list of
> proposed changes:
>
> 1) Add a caching mechanism.  The environment profile should get built
> once, and then a symlink to it should be created in $PWD and
> registered as a GC root.  This will, of course, require re-using some
> 'guix package' code to delete the profile.  For the sake of the rest
> of this post, let's say that a --cache flag does this, and a --update
> flag forces a rebuilding of the cached profile.

I always run 'guix environmeut guix' and have to wait for substitutes
before entering the shell, indeed caching the environment will save me
much time, but usually I don't mind that and was already used to the
"awlays being latest" behaviour...

I definitely want this feature too, so how about rename the current
implementation of 'environment' to 'guix shell', whose ad hoc behaviour
is similar to the 'nix-shell', and start a new implementation with this
persistent behaviour?

>
> 2) Make "ad-hoc" the default.  Remove the --ad-hoc flag and replace it
>with a --dependencies flag instead, reversing the current behavior.
>'guix environment --dependencies guix' would create a guix dev
>environment, for example.
>
> 3) Change how --load behaves.  Instead of evaluating a file and
> expected a package object in return, instead expect a
> <guix-environment> record.  This would provide a declarative way to
> specify the complete environment: which packages to pull in, whether
> to purify the environment (--pure), whether to run in a container
> (--container), whether to give the container network access
> (--network), etc.  Command line flags would take precedence over what
> is specified in the config file.  --load may only appear *once* in the
> command line args, whereas now it many appear N times.

I would expect a config file is mandatory and default to something like
'environment.scm' in the current directory, and a '--config' option can
be used to choose other location.  This 'environment' command don't have
many command line options (ad-hoc in nature), while the config file can
do a lot.

>
> 4) Make 'guix environment' with no other args operate like 'guix
> environment --cache --load=guix.scm'.  'guix.scm' is a placeholder
> name for whatever we decide the conventional name for an environment
> config should be.  Throwaway environments (such as 'guix environment
> ruby -- irb') would not have caching turned on by default, because
> that would quickly become a burden.

Yeah, split them into 2 commands will be more clear :-)
>
> 5) Add support for Shepherd services.  When an environment is created,
> a new Shepherd process is launched to handle all the necessary
> services.  For a web application this could be nginx, mysql, redis,
> etc.  This feature can be implemented later, even post 1.0, provided
> we agree upon using a <guix-environment> record type.

Yes, maybe launch the services with a subcommand like 'guix environment
up', so the shell entering 'guix environment' command can be run
multiple times.

>
> After all these changes, a Guix user should be able to jump into a new
> project, run 'guix environment' and be ready to go.  When they update
> Guix and want to refresh their environment, they would run 'guix
> environment --update'.
>
> Thoughts?

Indeed very useful, looking forward to it!

Re: The future of 'guix environment'

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1290 bytes --]

On Wed, Aug 30, 2017 at 11:11:33PM +0800, 宋文武 wrote:
> > 1) Add a caching mechanism.  The environment profile should get built
> > once, and then a symlink to it should be created in $PWD and
> > registered as a GC root.  This will, of course, require re-using some
> > 'guix package' code to delete the profile.  For the sake of the rest
> > of this post, let's say that a --cache flag does this, and a --update
> > flag forces a rebuilding of the cached profile.

I like that the results of `guix environment` are ephemeral by default.

> I always run 'guix environmeut guix' and have to wait for substitutes
> before entering the shell, indeed caching the environment will save me
> much time, but usually I don't mind that and was already used to the
> "awlays being latest" behaviour...
> 
> I definitely want this feature too, so how about rename the current
> implementation of 'environment' to 'guix shell', whose ad hoc behaviour
> is similar to the 'nix-shell', and start a new implementation with this
> persistent behaviour?

I agree. If we change the behaviour to leave files in the current
directory and register GC roots then we should use a new name or rename
the existing functionality. I'm in favor of using a new name for new
functionality.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: The future of 'guix environment'

[Ricardo Wurmus]

宋文武 <iyzsong@member.fsf.org> writes:

>> 5) Add support for Shepherd services.  When an environment is created,
>> a new Shepherd process is launched to handle all the necessary
>> services.  For a web application this could be nginx, mysql, redis,
>> etc.  This feature can be implemented later, even post 1.0, provided
>> we agree upon using a <guix-environment> record type.
>
> Yes, maybe launch the services with a subcommand like 'guix environment
> up', so the shell entering 'guix environment' command can be run
> multiple times.

AFAIU the proposal, that’s precisely what we don’t want.  Dave wants
users/developers to be able to specify the behaviour of the environment,
not just the packages that ought to be available.  I’d expect all
services to be killed upon leaving the environment, so we don’t need to
have a separate command for bringing them up.

It is not clear to me if it should be possible to “attach” to an
environment that is “active” in another shell session (like it is
possible with Docker containers).

@Dave: I pretty much agree with all of the proposed changes.  One thing
that may be lost, though, when using a new <guix-environment> record is
easy ad-hoc packaging.

Now I can use a guix.scm file both for building a Guix package and a
Docker image; I would expect that it would no longer be possible to
build a Guix package out of an environment file.  On the other hand,
building a more fully featured container image would be even simpler
with this change.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: The future of 'guix environment'

[Ludovic Courtès]

Hello!

Ricardo Wurmus <rekado@elephly.net> skribis:

> @Dave: I pretty much agree with all of the proposed changes.  One thing
> that may be lost, though, when using a new <guix-environment> record is
> easy ad-hoc packaging.
>
> Now I can use a guix.scm file both for building a Guix package and a
> Docker image; I would expect that it would no longer be possible to
> build a Guix package out of an environment file.  On the other hand,
> building a more fully featured container image would be even simpler
> with this change.

I think the new command could automatically convert <package> to
<environment>, such that you can still have a guix.scm file that defines
a package and can be passed both to ‘guix package -f’ and ‘guix
environment’.

Ludo’.

Re: The future of 'guix environment'

[Andreas Enge]

Hello,

On Wed, Aug 30, 2017 at 11:11:33PM +0800, 宋文武 wrote:
> I definitely want this feature too, so how about rename the current
> implementation of 'environment' to 'guix shell', whose ad hoc behaviour
> is similar to the 'nix-shell', and start a new implementation with this
> persistent behaviour?

I think we should not have too many commands to Guix, especially not with
similar behaviour, since this will make the tool too confusing. How about
adding command line options?

For instance:
1) Do not cache by default, cache when "--cache" is given.
2) Maybe even better, use "--cache=XXX" to cache under the name or number XXX.
3) Then later, add "--resume" or "--resume=XXX".

Or do something similar to "git stash", piling up environments on a stack
and popping or dropping them.

Andreas

Re: The future of 'guix environment'

[Thompson, David]

Thanks for the feedback, everyone.  I'm going to try to address what
everyone has been discussing.

宋文武, I agree with Andreas that making a new command is undesirable and
would just lead to confusion.  I think a single tool can support the
major use cases well.  I'd go so far as to say that I will have failed
if two commands are necessary.  We can always add subcommands to 'guix
environment' itself, if that turns out to be better for usability.  We
will see.

As for a 'guix environment up' command, I think that's worth thinking
about but I'm going to avoid thinking about it too much because we're
still quite a ways off from being able to make this happen.  My
general feeling is the same as Ricardo's, though, that 'guix
environment' should just spawn the services automatically and there
would have to be some notion of "attaching" to an existing session.
This topic deserves a whole separate thread when the time comes.

Ricardo, you are correct that we would lose the ability to use the
guix.scm file for both 'guix environment' and 'guix build'.  In
practice I don't actually use my guix.scm file this way, so I think
it's worth breaking, but maybe you (or someone else) actually uses
this and we should think more about it?

I wasn't very clear about whether ephemeral or cached would be the new
default.  I don't think there is one default for all cases, I think
it's more context sensitive.  When the user specifies --load, I think
caching should be the default because loading such a file means that
you are working on a project for which you'd like a persistent
environment.  When the user specifies ad-hoc packages, caching would
be disabled because most likely they are running a one-off job.  For
example, 'guix environment ruby -- irb' should spawn a Ruby REPL,
discarding the environment when the user exits.  'guix environment
--load=guix.scm' would check for a cached profile, use it if it is
there, otherwise build the profile and cache it for next time.  'guix
environment', without --load or any packages specified, would be
equivalent to 'guix environment --load=guix.scm'.  A little
"convention over configuration" will greatly improve usability, I
think.  There's really only 2 major use cases, and I want those to
work without having to fiddle with a bunch of command line flags.  All
the underlying flags will still be there to fiddle with for those who
are interested.

- Dave

Re: The future of 'guix environment'

[Christopher Allan Webber]

Thompson, David writes:

> Ricardo, you are correct that we would lose the ability to use the
> guix.scm file for both 'guix environment' and 'guix build'.  In
> practice I don't actually use my guix.scm file this way, so I think
> it's worth breaking, but maybe you (or someone else) actually uses
> this and we should think more about it?

I do use guix.scm for testing `guix build'.  Maybe I ma the only one
though.  It turns out to be very useful to find out whether or not I've
done something that borked my package that's not obvious because I have
some compiled .go file around in working directory or something.

> I wasn't very clear about whether ephemeral or cached would be the new
> default.  I don't think there is one default for all cases, I think
> it's more context sensitive.

I'm a bit confused by caching, but IMO maybe the right solution is
actually guix environment for "environment profiles".  Ie, I want to be
able to update my guix environment and if it turns out I made a mistake
easily roll back, just like I can with my user's profile.  Currently
you can use --root, but you don't have the generations support.

Re: The future of 'guix environment'

[Thompson, David]

On Thu, Aug 31, 2017 at 11:57 PM, Christopher Allan Webber
<cwebber@dustycloud.org> wrote:
> Thompson, David writes:
>
>> Ricardo, you are correct that we would lose the ability to use the
>> guix.scm file for both 'guix environment' and 'guix build'.  In
>> practice I don't actually use my guix.scm file this way, so I think
>> it's worth breaking, but maybe you (or someone else) actually uses
>> this and we should think more about it?
>
> I do use guix.scm for testing `guix build'.  Maybe I ma the only one
> though.  It turns out to be very useful to find out whether or not I've
> done something that borked my package that's not obvious because I have
> some compiled .go file around in working directory or something.

Ludovic's suggestion of supporting both package and environment
objects with --load will solve the issue.

>> I wasn't very clear about whether ephemeral or cached would be the new
>> default.  I don't think there is one default for all cases, I think
>> it's more context sensitive.
>
> I'm a bit confused by caching, but IMO maybe the right solution is
> actually guix environment for "environment profiles".  Ie, I want to be
> able to update my guix environment and if it turns out I made a mistake
> easily roll back, just like I can with my user's profile.  Currently
> you can use --root, but you don't have the generations support.

This is what caching would allow.

- Dave

Re: The future of 'guix environment'

[Ludovic Courtès]

Hi Chris,

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

> Thompson, David writes:
>
>> Ricardo, you are correct that we would lose the ability to use the
>> guix.scm file for both 'guix environment' and 'guix build'.  In
>> practice I don't actually use my guix.scm file this way, so I think
>> it's worth breaking, but maybe you (or someone else) actually uses
>> this and we should think more about it?
>
> I do use guix.scm for testing `guix build'.  Maybe I ma the only one
> though.  It turns out to be very useful to find out whether or not I've
> done something that borked my package that's not obvious because I have
> some compiled .go file around in working directory or something.

I use it too, occasionally!  :-)

I use a feature that maybe no one else uses, which is to have guix.scm
return a gexp.  I found it to be a relatively nice way to do
makefile-like things, like building the web site:

  http://git.savannah.gnu.org/cgit/guix/guix-artwork.git/tree/website/guix.scm

Ludo’.

Re: The future of 'guix environment'

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

>> I do use guix.scm for testing `guix build'.  Maybe I ma the only one
>> though.  It turns out to be very useful to find out whether or not I've
>> done something that borked my package that's not obvious because I have
>> some compiled .go file around in working directory or something.
>
> I use it too, occasionally!  :-)

Me too!  It's great to test if all is fine.  What I'm also using is your
git-file? hack

    https://git.dthompson.us/guile-sdl2.git/blob/HEAD:/guix.scm

does anyone see a nice way to put that function/functionaly in guix?

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: The future of 'guix environment'

[Thompson, David]

On Sun, Sep 3, 2017 at 3:15 AM, Jan Nieuwenhuizen <janneke@gnu.org> wrote:
> Ludovic Courtès writes:
>
>>> I do use guix.scm for testing `guix build'.  Maybe I ma the only one
>>> though.  It turns out to be very useful to find out whether or not I've
>>> done something that borked my package that's not obvious because I have
>>> some compiled .go file around in working directory or something.
>>
>> I use it too, occasionally!  :-)
>
> Me too!  It's great to test if all is fine.  What I'm also using is your
> git-file? hack
>
>     https://git.dthompson.us/guile-sdl2.git/blob/HEAD:/guix.scm
>
> does anyone see a nice way to put that function/functionaly in guix?

git-file? was snarfed from some internal Guix code that Ludovic wrote.
I would also like to see a public API for it.

- Dave

Re: The future of 'guix environment'

[Ludovic Courtès]

"Thompson, David" <dthompson2@worcester.edu> skribis:

> On Sun, Sep 3, 2017 at 3:15 AM, Jan Nieuwenhuizen <janneke@gnu.org> wrote:
>> Ludovic Courtès writes:
>>
>>>> I do use guix.scm for testing `guix build'.  Maybe I ma the only one
>>>> though.  It turns out to be very useful to find out whether or not I've
>>>> done something that borked my package that's not obvious because I have
>>>> some compiled .go file around in working directory or something.
>>>
>>> I use it too, occasionally!  :-)
>>
>> Me too!  It's great to test if all is fine.  What I'm also using is your
>> git-file? hack
>>
>>     https://git.dthompson.us/guile-sdl2.git/blob/HEAD:/guix.scm
>>
>> does anyone see a nice way to put that function/functionaly in guix?
>
> git-file? was snarfed from some internal Guix code that Ludovic wrote.
> I would also like to see a public API for it.

It’s ‘git-predicate’ in (guix git-download).

Ludo’.

Re: The future of 'guix environment'

[Andreas Enge]

Hello David,

thanks for your thoughts!

On Wed, Aug 30, 2017 at 09:22:13AM -0400, Thompson, David wrote:
> As an avid Emacs user, I want to do everything in Emacs, but if I want
> to use, say, M-x compile but in the context of my development
> environment, I have to wrap the command in 'guix environment' and now
> everything takes way longer because Guix wastes tons of time talking
> to the daemon to build a profile that I know already exists.   This
> goes hand in hand with the previous issue.  We need a cache.

From the remainder of your message, I get the impression that exactly one
environment is cached?

As Leo, I use "guix environment" only ephemerally, as "guix environment foo"
when I quickly want to build or work on package foo (which, most of the
time, equals "guix"...). So an option of no cache would be nice as well.

On the other hand, could there be several cached environments?
Numbered or otherwise identified, for people who alternate between two
different projects? Would this be an interesting use case?

> The default behavior of 'guix environment foo' creating an environment
> with the *dependencies* of the package 'foo' seemed like a good idea
> at the time (it's what nix-shell did, after all), but most of the time
> when I define throwaway environments on the command-line (as opposed
> to a guix.scm file) I *always* use the --ad-hoc flag.  Using the
> packages themselves rather than the dependencies should be the
> default.

Indeed, before getting used to it, I found this behaviour very counter-
intuitive, so I like the new defaults you suggest, with reversed command
line flags.

Andreas

Re: The future of 'guix environment'

[Jan Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 734 bytes --]

Thompson, David writes:

> Hi all, been awhile!

Hi David!

Great suggestions all!  I use guix environment heavily and like it a
lot.

> 4) Make 'guix environment' with no other args operate like 'guix
> environment --cache --load=guix.scm'.  'guix.scm' is a placeholder
> name for whatever we decide the conventional name for an environment
> config should be.

Wouldn't that be

    guix environment --cache --dependencies --load=guix.scm

I have been using an ugly hack to switch profiles (or environments; look
at $GUIX_ENVIRONMENT) in Emacs, see attached.  As discussed briefly on
irc yesterday are various ways this could go: use emacs-direnv, read the
manifest file in elisp, use guix repl to read it..etc

Greetings,
janneke


[-- Attachment #2: guix-environment.el --]
[-- Type: application/emacs-lisp, Size: 700 bytes --]

[-- Attachment #3: Type: text/plain, Size: 152 bytes --]


-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: The future of 'guix environment'

[Thompson, David]

On Thu, Aug 31, 2017 at 3:18 AM, Jan Nieuwenhuizen <janneke@gnu.org> wrote:
> Thompson, David writes:
>
>> Hi all, been awhile!
>
> Hi David!
>
> Great suggestions all!  I use guix environment heavily and like it a
> lot.

Thanks!

>> 4) Make 'guix environment' with no other args operate like 'guix
>> environment --cache --load=guix.scm'.  'guix.scm' is a placeholder
>> name for whatever we decide the conventional name for an environment
>> config should be.
>
> Wouldn't that be
>
>     guix environment --cache --dependencies --load=guix.scm

No, because guix.scm would no longer evaluate to a package object, but
an environment object.  Flags like --dependencies are for environments
defined on the fly at the command line.  I haven't yet decided if
--load should be mutually exclusive from all those other things, but I
think that should probably be the case.  Perhaps introducing
subcommands is the best thing UX-wise.  'guix environment load
guix.scm', 'guix environment shell direnv --dependencies ruby', and
'guix environment' would be a shorthand for 'guix environment load'.
This will take some fiddling to see.  I'm going to get the core
features implemented first, then work on the UX.

> I have been using an ugly hack to switch profiles (or environments; look
> at $GUIX_ENVIRONMENT) in Emacs, see attached.  As discussed briefly on
> irc yesterday are various ways this could go: use emacs-direnv, read the
> manifest file in elisp, use guix repl to read it..etc

I took a look at emacs-direnv and did not like it because it's not
fully implemented in elisp. You also need the direnv tool itself,
which is written in Go. No thanks!  I think the integration should
happen with emacs-guix, which could use the Guile REPL to read the
profile manifest, and send an sexp back to Emacs with key/value pairs
for the values of all environment variables.  From there it's easy to
use setenv and tweak exec-path in the case of $PATH.  Alex Kost would
know a lot better than I about how to get this done.

- Dave

Re: The future of 'guix environment'

[Jan Nieuwenhuizen]

Thompson, David writes:

>> Wouldn't that be
>>
>>     guix environment --cache --dependencies --load=guix.scm
>
> No, because guix.scm would no longer evaluate to a package object, but
> an environment object.

Ah, nice.  Thank for the explanation!

One thing I didn't see in your proposal is something like

    guix environment --manifest manifest.scm

It really bothers me that I can only declaritavely populate/maintain
named profiles.  Do you have an idea about that?

> I took a look at emacs-direnv and did not like it because it's not
> fully implemented in elisp. You also need the direnv tool itself,
> which is written in Go. No thanks!

No thanks here too.  A yoghurt language just to run a shell, ugh.


Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: The future of 'guix environment'

[Ricardo Wurmus]

Jan Nieuwenhuizen <janneke@gnu.org> writes:

> Thompson, David writes:
>> […] which is written in Go. No thanks!
>
> No thanks here too.  A yoghurt language just to run a shell, ugh.

The term “yoghurt […]” for software with bootstrapping problems will be
mainstream any day now :)

(But isn’t Go actually okay in terms of bootstrapping thanks to the GCC
implementation?)

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: The future of 'guix environment'

[Jan Nieuwenhuizen]

Ricardo Wurmus writes:

> The term “yoghurt […]” for software with bootstrapping problems will be
> mainstream any day now :)

:-)

> (But isn’t Go actually okay in terms of bootstrapping thanks to the GCC
> implementation?)

Yes it seems you're right, thanks.

gnu/packages/golang.scm says:

    As of go-1.5, go cannot be bootstrapped without go-1.4, so we need
    to use go-1.4 or gccgo-5.

So that sounds like more work/more steps, but still sweet milky

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: The future of 'guix environment'

[Thompson, David]

On Fri, Sep 1, 2017 at 7:50 AM, Jan Nieuwenhuizen <janneke@gnu.org> wrote:
>
> One thing I didn't see in your proposal is something like
>
>     guix environment --manifest manifest.scm
>
> It really bothers me that I can only declaritavely populate/maintain
> named profiles.  Do you have an idea about that?

Never thought about it before.  Maybe --load could support evaluating
to a package, a manifest, or an environment?  Seems like that might be
getting into crazy territory, but maybe not.

- Dave

Re: The future of 'guix environment'

[Ludovic Courtès]

Heya,

"Thompson, David" <dthompson2@worcester.edu> skribis:

> If you've followed along this far, great!  Now here's my list of
> proposed changes:
>
> 1) Add a caching mechanism.  The environment profile should get built
> once, and then a symlink to it should be created in $PWD and
> registered as a GC root.  This will, of course, require re-using some
> 'guix package' code to delete the profile.  For the sake of the rest
> of this post, let's say that a --cache flag does this, and a --update
> flag forces a rebuilding of the cached profile.

One way to put it is that ‘guix environment’ would be syntactic sugar
for ‘guix package -p $PWD/.guix-environment’, right?

This would make “guix environment” stateful: if you have something in
cache, that’s what you get (it could be old versions of “foo” and
“bar”), but if you don’t, you get the current versions of “foo” and
“bar”.  Is this what you have in mind?

I prefer the current stateless behavior, whereby “guix environment
--ad-hoc foo bar” always gives you the same environment, given a Guix
commit.  But maybe we can make “guix environment” (no arguments)
stateful, and keep “guix environment foo bar baz” stateless?

> 2) Make "ad-hoc" the default.  Remove the --ad-hoc flag and replace it
> with a --dependencies flag instead, reversing the current behavior.
> 'guix environment --dependencies guix' would create a guix dev
> environment, for example.

+1

> 3) Change how --load behaves.  Instead of evaluating a file and
> expected a package object in return, instead expect a
> <guix-environment> record.  This would provide a declarative way to
> specify the complete environment: which packages to pull in, whether
> to purify the environment (--pure), whether to run in a container
> (--container), whether to give the container network access
> (--network), etc.  Command line flags would take precedence over what
> is specified in the config file.  --load may only appear *once* in the
> command line args, whereas now it many appear N times.

+1, probably with automatic conversion of <package> to <environment>, as
I wrote in another message.

> 4) Make 'guix environment' with no other args operate like 'guix
> environment --cache --load=guix.scm'.  'guix.scm' is a placeholder
> name for whatever we decide the conventional name for an environment
> config should be.  Throwaway environments (such as 'guix environment
> ruby -- irb') would not have caching turned on by default, because
> that would quickly become a burden.

+1, except perhaps for --cache, not sure.

Also, I would prefer the convention to be “.guix.scm” (to avoid
confusion with the (guix) module, and to mimic existing conventions
followed by Travis and all that.)

Bonus: honor “.guix.json” (see <https://bugs.gnu.org/28251>) as one way
to get us closer to world domination.

> 5) Add support for Shepherd services.  When an environment is created,
> a new Shepherd process is launched to handle all the necessary
> services.  For a web application this could be nginx, mysql, redis,
> etc.  This feature can be implemented later, even post 1.0, provided
> we agree upon using a <guix-environment> record type.

Yup.

> After all these changes, a Guix user should be able to jump into a new
> project, run 'guix environment' and be ready to go.  When they update
> Guix and want to refresh their environment, they would run 'guix
> environment --update'.

These all sound like great proposals!

I’d rather not add a new command, which means we’ll have to agree to
break the “guix environment” CLI.  I think that’s OK, but we should
discuss it.

Thank you!

Ludo’.

Re: The future of 'guix environment'

[Thompson, David]

On Thu, Aug 31, 2017 at 11:16 AM, Ludovic Courtès <ludo@gnu.org> wrote:
> Heya,
>
> "Thompson, David" <dthompson2@worcester.edu> skribis:
>
>> If you've followed along this far, great!  Now here's my list of
>> proposed changes:
>>
>> 1) Add a caching mechanism.  The environment profile should get built
>> once, and then a symlink to it should be created in $PWD and
>> registered as a GC root.  This will, of course, require re-using some
>> 'guix package' code to delete the profile.  For the sake of the rest
>> of this post, let's say that a --cache flag does this, and a --update
>> flag forces a rebuilding of the cached profile.
>
> One way to put it is that ‘guix environment’ would be syntactic sugar
> for ‘guix package -p $PWD/.guix-environment’, right?

Yeah, I guess so.  I've never used the -p flag and don't know if it
should stay.  I'll try to preserve it and see how that goes!

> This would make “guix environment” stateful: if you have something in
> cache, that’s what you get (it could be old versions of “foo” and
> “bar”), but if you don’t, you get the current versions of “foo” and
> “bar”.  Is this what you have in mind?

Yes. For regularly hacked on project environments I think this is the
most useful behavior.  Because then, much like my regular user
profile, I can update at a time when I feel ready to do so, rather the
current situation where I'm forced to rebuild and deal with potential
breakage or long download/compile times.

> I prefer the current stateless behavior, whereby “guix environment
> --ad-hoc foo bar” always gives you the same environment, given a Guix
> commit.  But maybe we can make “guix environment” (no arguments)
> stateful, and keep “guix environment foo bar baz” stateless?

That's what I had in mind.  In my head there's two important cases:
the on-the-fly, stateless environment, and the persistent environment
that you update every now and then when you feel like living on the
edge.  'guix environment foo bar' (ad-hoc being the new default)
should absolutely be stateless, just like it is now.

>> 3) Change how --load behaves.  Instead of evaluating a file and
>> expected a package object in return, instead expect a
>> <guix-environment> record.  This would provide a declarative way to
>> specify the complete environment: which packages to pull in, whether
>> to purify the environment (--pure), whether to run in a container
>> (--container), whether to give the container network access
>> (--network), etc.  Command line flags would take precedence over what
>> is specified in the config file.  --load may only appear *once* in the
>> command line args, whereas now it many appear N times.
>
> +1, probably with automatic conversion of <package> to <environment>, as
> I wrote in another message.

Ah, this is a clever hack!  I will do that.  Thanks for suggesting it.

>> 4) Make 'guix environment' with no other args operate like 'guix
>> environment --cache --load=guix.scm'.  'guix.scm' is a placeholder
>> name for whatever we decide the conventional name for an environment
>> config should be.  Throwaway environments (such as 'guix environment
>> ruby -- irb') would not have caching turned on by default, because
>> that would quickly become a burden.
>
> +1, except perhaps for --cache, not sure.

I think I will convince you it's a good idea. ;)

> Also, I would prefer the convention to be “.guix.scm” (to avoid
> confusion with the (guix) module, and to mimic existing conventions
> followed by Travis and all that.)

Sure, that's fine with me, but FYI there are existing conventions of
using non-hidden files. Bundler uses 'Gemfile', Docker uses
'Dockerfile', Vagrant uses 'Vagrantfile', npm uses 'package.json',
etc.

> Bonus: honor “.guix.json” (see <https://bugs.gnu.org/28251>) as one way
> to get us closer to world domination.

Maybe someone else can add that after I do everything else. :)

> I’d rather not add a new command, which means we’ll have to agree to
> break the “guix environment” CLI.  I think that’s OK, but we should
> discuss it.

I agree.  We can discuss more when I have patches.  I will probably
have additional questions as I go.  The biggest unknown for me so far
is how environment profile generation management will work (if it
should even exist), but I'll get the basics sorted out first.

Thanks for your feedback!

- Dave

Re: The future of 'guix environment'

[Ludovic Courtès]

Howdy!

"Thompson, David" <dthompson2@worcester.edu> skribis:

> On Thu, Aug 31, 2017 at 11:16 AM, Ludovic Courtès <ludo@gnu.org> wrote:

[...]

>> This would make “guix environment” stateful: if you have something in
>> cache, that’s what you get (it could be old versions of “foo” and
>> “bar”), but if you don’t, you get the current versions of “foo” and
>> “bar”.  Is this what you have in mind?
>
> Yes. For regularly hacked on project environments I think this is the
> most useful behavior.  Because then, much like my regular user
> profile, I can update at a time when I feel ready to do so, rather the
> current situation where I'm forced to rebuild and deal with potential
> breakage or long download/compile times.

Agreed, that makes sense.

>> I prefer the current stateless behavior, whereby “guix environment
>> --ad-hoc foo bar” always gives you the same environment, given a Guix
>> commit.  But maybe we can make “guix environment” (no arguments)
>> stateful, and keep “guix environment foo bar baz” stateless?
>
> That's what I had in mind.  In my head there's two important cases:
> the on-the-fly, stateless environment, and the persistent environment
> that you update every now and then when you feel like living on the
> edge.  'guix environment foo bar' (ad-hoc being the new default)
> should absolutely be stateless, just like it is now.

Perfect.  So “guix environment” with no arguments would almost be a
wrapper around “guix package -p”, I think (with generations, upgrades,
etc.)

>> Also, I would prefer the convention to be “.guix.scm” (to avoid
>> confusion with the (guix) module, and to mimic existing conventions
>> followed by Travis and all that.)
>
> Sure, that's fine with me, but FYI there are existing conventions of
> using non-hidden files. Bundler uses 'Gemfile', Docker uses
> 'Dockerfile', Vagrant uses 'Vagrantfile', npm uses 'package.json',
> etc.

Oh right.  “Guixfile” then!

(Just kidding.)

Thanks!

Ludo’.

Re: The future of 'guix environment'

[Christopher Allan Webber]

Not going to reply in detail, but I'm a devoted user of `guix
environment' and I support all these changes!