all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* Named environments
@ 2021-09-10 23:06 Ryan Prior
  2021-09-11  8:59 ` Domagoj Stolfa
                   ` (2 more replies)
  0 siblings, 3 replies; 10+ messages in thread
From: Ryan Prior @ 2021-09-10 23:06 UTC (permalink / raw)
  To: guix-devel@gnu.org

[-- Attachment #1: Type: text/plain, Size: 821 bytes --]

Hey Guix.

I've been thinking lately it would be convenient to create certain uniquely named execution environments on my machine. For example, I might have one set up with dependencies for my Python webapp & environment variables set to autoconnect to a Postgres server. I might have another that's got test dependencies and is containerized, such that it can only access the network & not the rest of my filesystem. Suppose I name these two "webapp" and "test" respectively.

I picture running eg `guix env @webapp -- uvicorn main:app` to start my server, then `guix env @test -- pytest` to run my tests.

I might write a wrapper in some scripting language that sets up this kind of system. Would anybody else be interested in using such a thing? Would it make sense to integrate this capability into Guix itself?

Ryan

[-- Attachment #2: Type: text/html, Size: 1205 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Named environments
  2021-09-10 23:06 Named environments Ryan Prior
@ 2021-09-11  8:59 ` Domagoj Stolfa
  2021-09-12 19:33 ` Sarah Morgensen
  2021-09-13 15:48 ` pinoaffe
  2 siblings, 0 replies; 10+ messages in thread
From: Domagoj Stolfa @ 2021-09-11  8:59 UTC (permalink / raw)
  To: Ryan Prior; +Cc: guix-devel@gnu.org

[-- Attachment #1: Type: text/plain, Size: 372 bytes --]

Hi Ryan:

> I might write a wrapper in some scripting language that sets up this kind of system. Would anybody else be interested in using such a thing? Would it make sense to integrate this capability into Guix itself?

I've been using something like that with shell scripts and manifests. I think
it would be great to have such functionality in Guix itself!

--
Domagoj

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Named environments
  2021-09-10 23:06 Named environments Ryan Prior
  2021-09-11  8:59 ` Domagoj Stolfa
@ 2021-09-12 19:33 ` Sarah Morgensen
  2021-09-13  8:08   ` zimoun
  2021-10-06 13:29   ` Ludovic Courtès
  2021-09-13 15:48 ` pinoaffe
  2 siblings, 2 replies; 10+ messages in thread
From: Sarah Morgensen @ 2021-09-12 19:33 UTC (permalink / raw)
  To: Ryan Prior; +Cc: guix-devel


Ryan Prior <rprior@protonmail.com> writes:

> Hey Guix.
>
> I've been thinking lately it would be convenient to create certain uniquely
> named execution environments on my machine. For example, I might have one
> set up with dependencies for my Python webapp & environment variables set to
> autoconnect to a Postgres server. I might have another that's got test
> dependencies and is containerized, such that it can only access the network
> & not the rest of my filesystem. Suppose I name these two "webapp" and
> "test" respectively.
>
> I picture running eg `guix env @webapp -- uvicorn main:app` to start my
> server, then `guix env @test -- pytest` to run my tests.
>
> I might write a wrapper in some scripting language that sets up this kind of
> system. Would anybody else be interested in using such a thing? Would it
> make sense to integrate this capability into Guix itself?
>
>Ryan

I like this idea!  (And adding a built-in alias from "environment" to "env"
might be worth it, too.)

I actually use an ad-hoc version of this in my ~/.bashrc, using aliases. For
example,

  alias geg='guix environment guix --pure --ad-hoc nano git git:send-email less help2man strace nss-certs'

This also lets me add on any ad-hoc packages at the end, like

  geg dejagnu expect

However, an actual `guix env' could do a lot more!  Like you allude to, being
able to essentially save the options of a `guix environment' invocation would
be great:

  $ guix env --edit @test <lots of packages and environment options>
  $ guix env @test -- pytest

But also imagine portable environments (used, for example, for project dev
environments, checked in with the source):

--8<---------------cut here---------------start------------->8---
$ guix env --export @myapp-test --pure -C -f guix.scm --ad-hoc strace coreutils

(environment "@myapp-test"
 (load '("guix.scm"))
 (ad-hoc-packages '("strace" "coreutils" "findutils")) 
 (flags '(pure container)))
--8<---------------cut here---------------end--------------->8---

Of course, this isn't entirely reproducible, as packages could change as you
update your Guix, even if you give a version spec (the same issue that exists
for manifests).  So, we could pin the channel used to make the environment:

--8<---------------cut here---------------start------------->8---
$ guix env --export @myapp-test --pin-channels ...

(environment "@myapp-test"
 (channels (list (channel (name 'guix)
                          (url ...)
                          (commit ...)))
 (load '("guix.scm"))
 (ad-hoc-packages '("strace" "coreutils" "findutils")) 
 (flags '(pure container))))
--8<---------------cut here---------------end--------------->8---

which, when evaluated, would internally do what `guix time-machine --channels
file' does.

These environments could then be used like:

  $ guix env -e environment.scm @myapp-test

but that's quite long... perhaps we could allow `guix env' to automatically
read a "guix.scm" or "environment.scm" file in the current directory, and make
the environment definitions within available?

...so I definitely think there's room for such a feature in Guix :)

--
Sarah


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Named environments
  2021-09-12 19:33 ` Sarah Morgensen
@ 2021-09-13  8:08   ` zimoun
  2021-10-06 13:29   ` Ludovic Courtès
  1 sibling, 0 replies; 10+ messages in thread
From: zimoun @ 2021-09-13  8:08 UTC (permalink / raw)
  To: Sarah Morgensen, Ryan Prior; +Cc: guix-devel

Hi,

On Sun, 12 Sep 2021 at 12:33, Sarah Morgensen <iskarian@mgsn.dev> wrote:
> Ryan Prior <rprior@protonmail.com> writes:

>> I've been thinking lately it would be convenient to create certain uniquely
>> named execution environments on my machine. For example, I might have one
>> set up with dependencies for my Python webapp & environment variables set to
>> autoconnect to a Postgres server. I might have another that's got test
>> dependencies and is containerized, such that it can only access the network
>> & not the rest of my filesystem. Suppose I name these two "webapp" and
>> "test" respectively.
>>
>> I picture running eg `guix env @webapp -- uvicorn main:app` to start my
>> server, then `guix env @test -- pytest` to run my tests.
>>
>> I might write a wrapper in some scripting language that sets up this kind of
>> system. Would anybody else be interested in using such a thing? Would it
>> make sense to integrate this capability into Guix itself?
>
> I like this idea!  (And adding a built-in alias from "environment" to "env"
> might be worth it, too.)
>
> I actually use an ad-hoc version of this in my ~/.bashrc, using aliases. For
> example,
>
>   alias geg='guix environment guix --pure --ad-hoc nano git git:send-email less help2man strace nss-certs'
>
> This also lets me add on any ad-hoc packages at the end, like
>
>   geg dejagnu expect

You might be interested by GUIX_EXTENSIONS_PATH which is somehow
documented by this thread: :-)

<https://yhetil.org/guix/86k0sekkj8.fsf@gmail.com/>

It is a not-enough known feature but really handy to:

 1) define your own flavour, e.g., “guix env --options you --like“
 2) experiment for a new subcommand

Once a subcommand is in Guix, it is really hard to make a CLI change
so it seems better to experiment before, IMHO. :-)

For an instance of this last claim: <http://issues.guix.gnu.org/38529>.

> But also imagine portable environments (used, for example, for project dev
> environments, checked in with the source):
>
> --8<---------------cut here---------------start------------->8---
> $ guix env --export @myapp-test --pure -C -f guix.scm --ad-hoc strace coreutils
>
> (environment "@myapp-test"
>  (load '("guix.scm"))
>  (ad-hoc-packages '("strace" "coreutils" "findutils"))
>  (flags '(pure container)))
> --8<---------------cut here---------------end--------------->8---

This kind of ideas is exposed here:

<https://yhetil.org/guix/877dvn10ro.fsf@dustycloud.org/>

or here and there:

<https://lists.gnu.org/archive/html/guix-devel/2016-07/msg00120.html>
<https://lists.gnu.org/archive/html/help-guix/2018-01/msg00056.html>

Thanks for the revival. :-)


> Of course, this isn't entirely reproducible, as packages could change as you
> update your Guix, even if you give a version spec (the same issue that exists
> for manifests).  So, we could pin the channel used to make the environment:
>
> --8<---------------cut here---------------start------------->8---
> $ guix env --export @myapp-test --pin-channels ...
>
> (environment "@myapp-test"
>  (channels (list (channel (name 'guix)
>                           (url ...)
>                           (commit ...)))
>  (load '("guix.scm"))
>  (ad-hoc-packages '("strace" "coreutils" "findutils"))
>  (flags '(pure container))))
> --8<---------------cut here---------------end--------------->8---
>
> which, when evaluated, would internally do what `guix time-machine --channels
> file' does.

I think the correct workflow is:

  $ guix describe -f channels > channels.scm
  $ guix time-machine -C channels.scm -- subcommand --options


All the best,
simon


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Named environments
  2021-09-10 23:06 Named environments Ryan Prior
  2021-09-11  8:59 ` Domagoj Stolfa
  2021-09-12 19:33 ` Sarah Morgensen
@ 2021-09-13 15:48 ` pinoaffe
  2021-09-14  2:32   ` Ryan Prior
  2 siblings, 1 reply; 10+ messages in thread
From: pinoaffe @ 2021-09-13 15:48 UTC (permalink / raw)
  To: Ryan Prior; +Cc: guix-devel

Hey guix,

to me, this sounds very similar to the features offered by profiles,
which differences do you envision?

blessings,
pinoaffe

Ryan Prior writes:

> Hey Guix.
>
> I've been thinking lately it would be convenient to create certain uniquely named execution environments on my machine. For example, I might have one set up with dependencies for my Python webapp & environment variables set to autoconnect to a Postgres server. I might have another that's got test dependencies and is containerized, such that it can only access the network & not the rest of my filesystem. Suppose I name these two "webapp" and "test" respectively.
>
> I picture running eg `guix env @webapp -- uvicorn main:app` to start my server, then `guix env @test -- pytest` to run my tests.
>
> I might write a wrapper in some scripting language that sets up this kind of system. Would anybody else be interested in using such a thing? Would it make sense to integrate this capability into Guix itself?
>
> Ryan



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Named environments
  2021-09-13 15:48 ` pinoaffe
@ 2021-09-14  2:32   ` Ryan Prior
  2021-09-14  6:27     ` Lars-Dominik Braun
  2021-09-14  6:30     ` zimoun
  0 siblings, 2 replies; 10+ messages in thread
From: Ryan Prior @ 2021-09-14  2:32 UTC (permalink / raw)
  To: pinoaffe; +Cc: guix-devel

On Monday, September 13th, 2021 at 3:48 PM, pinoaffe <pinoaffe@airmail.cc> wrote:

> this sounds very similar to the features offered by profiles

Agreed, and maybe it's good to unify these things or be more explicit about how they're different?

What I see as the defining characteristic of a profile, is that it exists in a certain directory with a script (etc/profile) & structure to allow an end-user to activate it.

It lacks some things I associate with the concept of named environments:

- There is afaik no definitive list of profiles on your machine. I know "guix gc" knows somethings, but "gc --show-roots" doesn't give me useful info here. If I give names to "named environments" then that implies there's some registry of names which I can easily query to refer to a specific environment. This is important for integration with tools where you want to refer to a specific environment (profile?) by a shorthand name.

- I can easily update environments using my text editor, because they're based on manifests. I know you can export & import profiles, but the API to update profiles is basically imperative. Beyond that, it doesn't seem like you can check a profile into source control, but I'd very much like to check manifests for named environments into git to share them with people. Maybe this can just be bridged and we can get some kind of profiles-as-code concept going, as well as an imperative API for updating manifests (as Sarah Morgensen proposed IIRC with her "guix env --edit" command)

- Profiles don't seem to have certain desirable features that Guix environments do, such as pure execution, containerization with or without access to network or certain parts of the filesystem, etc. Maybe my understanding here is superficial and you can actually do this with profiles after all?


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Named environments
  2021-09-14  2:32   ` Ryan Prior
@ 2021-09-14  6:27     ` Lars-Dominik Braun
  2021-09-14  9:58       ` Pjotr Prins
  2021-09-14  6:30     ` zimoun
  1 sibling, 1 reply; 10+ messages in thread
From: Lars-Dominik Braun @ 2021-09-14  6:27 UTC (permalink / raw)
  To: Ryan Prior; +Cc: guix-devel

Hi,

> - Profiles don't seem to have certain desirable features that Guix environments do, such as pure execution, containerization with or without access to network or certain parts of the filesystem, etc. Maybe my understanding here is superficial and you can actually do this with profiles after all?
you can actually start a `guix environment` from an existing profile
using the -p switch and enjoy all of `guix environment`’s advantages
already.

Cheers,
Lars



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Named environments
  2021-09-14  2:32   ` Ryan Prior
  2021-09-14  6:27     ` Lars-Dominik Braun
@ 2021-09-14  6:30     ` zimoun
  1 sibling, 0 replies; 10+ messages in thread
From: zimoun @ 2021-09-14  6:30 UTC (permalink / raw)
  To: Ryan Prior, pinoaffe; +Cc: guix-devel

Hi Ryan,

On Tue, 14 Sep 2021 at 02:32, Ryan Prior <rprior@protonmail.com> wrote:
> On Monday, September 13th, 2021 at 3:48 PM, pinoaffe <pinoaffe@airmail.cc> wrote:
>
>> this sounds very similar to the features offered by profiles
>
> Agreed, and maybe it's good to unify these things or be more explicit
> about how they're different?

“guix environment” is a temporary profile. ;-)

--8<---------------cut here---------------start------------->8---
$ guix environment --ad-hoc hello
$ echo $GUIX_ENVIRONMENT
/gnu/store/lw9x5aimyqcq5iazj786fv7q5l3h0syk-profile
--8<---------------cut here---------------end--------------->8---

And see the option ’--profile’ and ’--root’.

Years ago, it had been discussed a “guix profile” subcommand in order to
help the management.  Maybe it could be resumed.

Also note what is really missing, IMHO, is the feature to
“containerized” some apps.

> What I see as the defining characteristic of a profile, is that it
> exists in a certain directory with a script (etc/profile) & structure
> to allow an end-user to activate it.

Well, a proof-of-concept attempt of such feature could be done with
a script and “guix repl”, then using “GUIX_EXTENSIONS_PATH”, you could
type:

  guix profile load

As a starting point, see
<https://lists.gnu.org/archive/html/guix-devel/2020-04/msg00477.html>.


> It lacks some things I associate with the concept of named environments:
>
> - There is afaik no definitive list of profiles on your machine. I

“guix package --list-profiles” lists all the user profiles.
Running the same as root lists all the profiles.
Do you need more?

> know "guix gc" knows somethings, but "gc --show-roots" doesn't give me
> useful info here. If I give names to "named environments" then that
> implies there's some registry of names which I can easily query to
> refer to a specific environment. This is important for integration
> with tools where you want to refer to a specific environment
> (profile?) by a shorthand name.
>
>
> - I can easily update environments using my text editor, because
> they're based on manifests. I know you can export & import profiles,
> but the API to update profiles is basically imperative. Beyond that,

I do not understand what means «update profiles is basically
imperative».  Somehow, a profile is a list of transactions; the way to
populate such transaction can be “imperative” or “declarative” but I
miss how the transaction itself could be “imperative” or “declarative”.
Well, from my understanding. :-)

> it doesn't seem like you can check a profile into source control, but
> I'd very much like to check manifests for named environments into git

You are speaking about the manifest.scm file, right?  Not the
internal etc/manifest file.

From my understanding, the way to share environment is to share this
manifest.scm file and the channels.scm file.  However, as explained
earlier or elsewhere, it miss “something” to easily write manifest.scm
file where some apps run in container.  Currently, it is:

  guix time-machine -C channels.scm \
       environment -m manifest.scm \
       -C -E <blablabla>

it could be nice to be able to easily declare all the “-C -E <blabla>”
directly in the manifest.scm file.

> to share them with people. Maybe this can just be bridged and we can
> get some kind of profiles-as-code concept going, as well as an
> imperative API for updating manifests (as Sarah Morgensen proposed
> IIRC with her "guix env --edit" command)

Once a profile, you can use ’--export-manifest’.

--8<---------------cut here---------------start------------->8---
$ guix environment --ad-hoc hello
$ echo $GUIX_ENVIRONMENT
/gnu/store/lw9x5aimyqcq5iazj786fv7q5l3h0syk-profile

$ exit
exit

$ guix package --export-manifest -p /gnu/store/lw9x5aimyqcq5iazj786fv7q5l3h0syk-profile
;; This "manifest" file can be passed to 'guix package -m' to reproduce
;; the content of your profile.  This is "symbolic": it only specifies
;; package names.  To reproduce the exact same profile, you also need to
;; capture the channels being used, as returned by "guix describe".
;; See the "Replicating Guix" section in the manual.

(specifications->manifest (list "hello"))
--8<---------------cut here---------------end--------------->8---

You cannot edit on-the-fly the manifest.  It would break the declarative
approach, IMHO.  Well, from my understanding, you cannot have a link
between the manifest.scm file which is used to instantiate a profile and
the profile itself.  It is not a 1:1 map.  Once the profile (named
environment or not :-)), you can only recreate an approximation of a
manifest.scm file.

And from my understanding, the option “-C -E <blabla>“ are not encoded
in the profile.  Maybe something is missing in this area.


> - Profiles don't seem to have certain desirable features that Guix
> environments do, such as pure execution, containerization with or
> without access to network or certain parts of the filesystem,
> etc. Maybe my understanding here is superficial and you can actually
> do this with profiles after all? 

I agree, something is missing here.

All the best,
simon


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Named environments
  2021-09-14  6:27     ` Lars-Dominik Braun
@ 2021-09-14  9:58       ` Pjotr Prins
  0 siblings, 0 replies; 10+ messages in thread
From: Pjotr Prins @ 2021-09-14  9:58 UTC (permalink / raw)
  To: Lars-Dominik Braun; +Cc: guix-devel

On Tue, Sep 14, 2021 at 08:27:10AM +0200, Lars-Dominik Braun wrote:
> Hi,
> 
> > - Profiles don't seem to have certain desirable features that Guix environments do, such as pure execution, containerization with or without access to network or certain parts of the filesystem, etc. Maybe my understanding here is superficial and you can actually do this with profiles after all?
> you can actually start a `guix environment` from an existing profile
> using the -p switch and enjoy all of `guix environment`’s advantages
> already.

Yes, even with manifests. Typical is to create a profile -p
~/opt/emacs that contains all my loved emacs extensions or, ruby 2.6
with some libs

  guix package -i ruby@2.6.x ruby-xml ruby-sinatra -p ~/opt/ruby2.6

and

  guix pull -p ~/opt/guix-latest 
  
is another favourite now.

Pj.


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Named environments
  2021-09-12 19:33 ` Sarah Morgensen
  2021-09-13  8:08   ` zimoun
@ 2021-10-06 13:29   ` Ludovic Courtès
  1 sibling, 0 replies; 10+ messages in thread
From: Ludovic Courtès @ 2021-10-06 13:29 UTC (permalink / raw)
  To: Sarah Morgensen; +Cc: guix-devel

Hi!

Sarah Morgensen <iskarian@mgsn.dev> skribis:

> However, an actual `guix env' could do a lot more!  Like you allude to, being
> able to essentially save the options of a `guix environment' invocation would
> be great:
>
>   $ guix env --edit @test <lots of packages and environment options>
>   $ guix env @test -- pytest

Perhaps not as convenient (although that’s a matter of taste), but that
sounds similar to:

  guix environment -m test-manifest.scm -- pytest

WDYT?  Perhaps there’s a need for a more lightweight CLI?

> But also imagine portable environments (used, for example, for project dev
> environments, checked in with the source):
>
> $ guix env --export @myapp-test --pure -C -f guix.scm --ad-hoc strace coreutils
>
> (environment "@myapp-test"
>  (load '("guix.scm"))
>  (ad-hoc-packages '("strace" "coreutils" "findutils")) 
>  (flags '(pure container)))

Like zimoun write, we could make ‘--export-profile’ somehow available to
‘guix environment’.

> Of course, this isn't entirely reproducible, as packages could change as you
> update your Guix, even if you give a version spec (the same issue that exists
> for manifests).  So, we could pin the channel used to make the environment:
>
> $ guix env --export @myapp-test --pin-channels ...
>
> (environment "@myapp-test"
>  (channels (list (channel (name 'guix)
>                           (url ...)
>                           (commit ...)))
>  (load '("guix.scm"))
>  (ad-hoc-packages '("strace" "coreutils" "findutils")) 
>  (flags '(pure container))))
>
> which, when evaluated, would internally do what `guix time-machine --channels
> file' does.

While it’s tempting to do both at once, I think channels and
environments should remain separate things, at least for now, because
the implications and cost differ significantly.

Thanks,
Ludo’.


^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2021-10-06 13:30 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-10 23:06 Named environments Ryan Prior
2021-09-11  8:59 ` Domagoj Stolfa
2021-09-12 19:33 ` Sarah Morgensen
2021-09-13  8:08   ` zimoun
2021-10-06 13:29   ` Ludovic Courtès
2021-09-13 15:48 ` pinoaffe
2021-09-14  2:32   ` Ryan Prior
2021-09-14  6:27     ` Lars-Dominik Braun
2021-09-14  9:58       ` Pjotr Prins
2021-09-14  6:30     ` zimoun

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.