Feature requests

Feature requests

[Joshua Marshall]

[-- Attachment #1: Type: text/plain, Size: 1746 bytes --]

Hello all,

I was told in IRC to post my possibly unreasonable feature requests here.

I've been thinking more on what Guix might be able to do, and what would
make it more useful for a few of my past jobs.  I'd like to see it take on
the ability to have a per-installation target cgroup, network namespace,
and filesystem chroot settings set with defaults which are overridable at
invocation.  In this way, a user could install and use packages with
mutually incompatible dependencies (I talked about this with a few people
on IRC) like what happens with python.  If this kind of functionality were
added, it would largely supplant Docker, virtualenv, pip, poetry, apk,
pacman, and probably a few other tools at my company which are there just
to handle this kind of frailness.  From this, I could also see an entry
point to adding build module support to start to replace tools like Make,
CMake, Meson, Bazel, and so on.

These expand the scope of Guix quite a bit, but I think these are needed
for it to really feel logically complete.  Does all this make sense?

-- 



Please be advised that this email may contain confidential information. 
If you are not the intended recipient, please notify us by email by 
replying to the sender and delete this message. The sender disclaims that 
the content of this email constitutes an offer to enter into, or the 
acceptance of, any agreement; provided that the foregoing does not 
invalidate the binding effect of any digital or other electronic 
reproduction of a manual signature that is included in any attachment.


 
<https://twitter.com/arroyo_networks>   
<https://www.linkedin.com/company/arroyo-networks>   
<https://www.github.com/ArroyoNetworks>

[-- Attachment #2: Type: text/html, Size: 2783 bytes --]

Re: Feature requests

[Julien Lepiller]

[-- Attachment #1: Type: text/plain, Size: 2632 bytes --]

Hi!

I'm not sure what you mean when you talk about incompatible packages, maybe you could give a concrete example? I don't think there's anyching that couldn't go to the store at least… So you can always create separate profiles.

That said, I think people are working on improving the network support in guix environment containers, and I think it's a good thing :). I'm not sure about cgroups, but our environments already provide quite a bit of isolation. Have you tried "guix environment --ad-hoc python --container" for instance? There are more options to control what parts of the file system is available inside the container.

Le 22 mars 2019 18:47:19 GMT+01:00, Joshua Marshall <j.marshall@arroyo.io> a écrit :
>Hello all,
>
>I was told in IRC to post my possibly unreasonable feature requests
>here.
>
>I've been thinking more on what Guix might be able to do, and what
>would
>make it more useful for a few of my past jobs.  I'd like to see it take
>on
>the ability to have a per-installation target cgroup, network
>namespace,
>and filesystem chroot settings set with defaults which are overridable
>at
>invocation.  In this way, a user could install and use packages with
>mutually incompatible dependencies (I talked about this with a few
>people
>on IRC) like what happens with python.  If this kind of functionality
>were
>added, it would largely supplant Docker, virtualenv, pip, poetry, apk,
>pacman, and probably a few other tools at my company which are there
>just
>to handle this kind of frailness.  From this, I could also see an entry
>point to adding build module support to start to replace tools like
>Make,
>CMake, Meson, Bazel, and so on.
>
>These expand the scope of Guix quite a bit, but I think these are
>needed
>for it to really feel logically complete.  Does all this make sense?
>
>-- 
>
>
>
>Please be advised that this email may contain confidential information.
>
>If you are not the intended recipient, please notify us by email by 
>replying to the sender and delete this message. The sender disclaims
>that 
>the content of this email constitutes an offer to enter into, or the 
>acceptance of, any agreement; provided that the foregoing does not 
>invalidate the binding effect of any digital or other electronic 
>reproduction of a manual signature that is included in any attachment.
>
>
> 
><https://twitter.com/arroyo_networks>   
><https://www.linkedin.com/company/arroyo-networks>   
><https://www.github.com/ArroyoNetworks>

-- 
Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma brièveté.

[-- Attachment #2: Type: text/html, Size: 3870 bytes --]

Re: Feature requests

[Joshua Marshall]

[-- Attachment #1: Type: text/plain, Size: 4414 bytes --]

I'm still getting my feet wet as it were.  As a more relatable example,
lets use glibc.  Say an application developed in 2000 had bug compatibility
with glibc at that time.  Then say that bug is fixed circa 2005 and another
introduced.  Then have in 2010 an application made with bug compatibility
for the 2005 version.  Now, if both of these applications were co-installed
at most one could work due to bug compatibility.  Now I'm not saying this
is good practice or even common, but this type of thing happens at my work
frequently enough that we have a whole toolset dedicated to managing just
this problem.  It gets far more severe in the python world.  To cope with
such poorly behaved programs, greater levels of isolation are needed, like
each individual thing running in its own Docker container.  But sometimes
even this isn't enough as the dependencies will have a similar problem and
so such isolation measures need to be recursively applied.  It gets messy.

On Fri, Mar 22, 2019 at 6:30 PM Julien Lepiller <julien@lepiller.eu> wrote:

> Hi!
>
> I'm not sure what you mean when you talk about incompatible packages,
> maybe you could give a concrete example? I don't think there's anyching
> that couldn't go to the store at least… So you can always create separate
> profiles.
>
> That said, I think people are working on improving the network support in
> guix environment containers, and I think it's a good thing :). I'm not sure
> about cgroups, but our environments already provide quite a bit of
> isolation. Have you tried "guix environment --ad-hoc python --container"
> for instance? There are more options to control what parts of the file
> system is available inside the container.
>
> Le 22 mars 2019 18:47:19 GMT+01:00, Joshua Marshall <j.marshall@arroyo.io>
> a écrit :
>>
>> Hello all,
>>
>> I was told in IRC to post my possibly unreasonable feature requests here.
>>
>> I've been thinking more on what Guix might be able to do, and what would
>> make it more useful for a few of my past jobs.  I'd like to see it take on
>> the ability to have a per-installation target cgroup, network namespace,
>> and filesystem chroot settings set with defaults which are overridable at
>> invocation.  In this way, a user could install and use packages with
>> mutually incompatible dependencies (I talked about this with a few people
>> on IRC) like what happens with python.  If this kind of functionality were
>> added, it would largely supplant Docker, virtualenv, pip, poetry, apk,
>> pacman, and probably a few other tools at my company which are there just
>> to handle this kind of frailness.  From this, I could also see an entry
>> point to adding build module support to start to replace tools like Make,
>> CMake, Meson, Bazel, and so on.
>>
>> These expand the scope of Guix quite a bit, but I think these are needed
>> for it to really feel logically complete.  Does all this make sense?
>>
>>
>> Please be advised that this email may contain confidential information.
>> If you are not the intended recipient, please notify us by email by
>> replying to the sender and delete this message. The sender disclaims that
>> the content of this email constitutes an offer to enter into, or the
>> acceptance of, any agreement; provided that the foregoing does not
>> invalidate the binding effect of any digital or other electronic
>> reproduction of a manual signature that is included in any attachment.
>>
>> <https://twitter.com/arroyo_networks>
>> <https://www.linkedin.com/company/arroyo-networks>
>> <https://www.github.com/ArroyoNetworks>
>>
>
> --
> Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma
> brièveté.
>

-- 



Please be advised that this email may contain confidential information. 
If you are not the intended recipient, please notify us by email by 
replying to the sender and delete this message. The sender disclaims that 
the content of this email constitutes an offer to enter into, or the 
acceptance of, any agreement; provided that the foregoing does not 
invalidate the binding effect of any digital or other electronic 
reproduction of a manual signature that is included in any attachment.


 
<https://twitter.com/arroyo_networks>   
<https://www.linkedin.com/company/arroyo-networks>   
<https://www.github.com/ArroyoNetworks>

[-- Attachment #2: Type: text/html, Size: 6849 bytes --]

Re: Feature requests

[Julien Lepiller]

[-- Attachment #1: Type: text/plain, Size: 7016 bytes --]

Mh… so I'm still not sure if that's an actual issue. You can always install two software that rely on different versions of a library. I guess in your example it only means that we need two libc packages, right? As long as you're able to define a package recipe for everything you want, using different versions of the same package is fine, because they are different inputs, so they have a different store path.

A package built with glibc 2.10 well reference that old libc and never know about the newer ones even if they're installed in the same profile, as long as the libc is not a propagated-input.

I still don't see an incompatibility that's not solved by guix. With python packages however that's very different because inputs are propagated. If python could reference its dependencies directly, it would have been just fine, but indeed two incompatible dependencies will end up in your profile. We can wrap python software already though, so it ignores the environment and picks only the dependencies it was built with. That's not always the best solution though. Maybe you can give as a concrete example for the python case? Then we will be able to help you better and come up with the right solution for your problems :)

From what you describe I still think that guix environments are what you want: if you have python-foo and python-bar that both depend on incompatible versions of python-baz, you can create an environment for python-foo:

guix environment --ad-hoc python python-foo --pure

In which you can use python-foo but not python-bar, and a similar command will give you a similar environment where you can run python-bar but not python-foo.

If you suppose these python packages provide a foo and a bar command, you could also do:

alias bar=guix environment --ad-hoc python python-bar --pure -- bar

And the same for foo, and you would have isolated environments. You can also use that to control what python-foo has access to at runtime. For instance if it can use python-foobar at runtime, you would do:

alias foo=guix environment --ad-hoc python python-foo python-foobar --pure -- foo

And the foo command now has access to its runtime dependency (if it wasn't already declared at build time).

HTH

Le 23 mars 2019 15:01:24 GMT+01:00, Joshua Marshall <j.marshall@arroyo.io> a écrit :
>I'm still getting my feet wet as it were.  As a more relatable example,
>lets use glibc.  Say an application developed in 2000 had bug
>compatibility
>with glibc at that time.  Then say that bug is fixed circa 2005 and
>another
>introduced.  Then have in 2010 an application made with bug
>compatibility
>for the 2005 version.  Now, if both of these applications were
>co-installed
>at most one could work due to bug compatibility.  Now I'm not saying
>this
>is good practice or even common, but this type of thing happens at my
>work
>frequently enough that we have a whole toolset dedicated to managing
>just
>this problem.  It gets far more severe in the python world.  To cope
>with
>such poorly behaved programs, greater levels of isolation are needed,
>like
>each individual thing running in its own Docker container.  But
>sometimes
>even this isn't enough as the dependencies will have a similar problem
>and
>so such isolation measures need to be recursively applied.  It gets
>messy.
>
>On Fri, Mar 22, 2019 at 6:30 PM Julien Lepiller <julien@lepiller.eu>
>wrote:
>
>> Hi!
>>
>> I'm not sure what you mean when you talk about incompatible packages,
>> maybe you could give a concrete example? I don't think there's
>anyching
>> that couldn't go to the store at least… So you can always create
>separate
>> profiles.
>>
>> That said, I think people are working on improving the network
>support in
>> guix environment containers, and I think it's a good thing :). I'm
>not sure
>> about cgroups, but our environments already provide quite a bit of
>> isolation. Have you tried "guix environment --ad-hoc python
>--container"
>> for instance? There are more options to control what parts of the
>file
>> system is available inside the container.
>>
>> Le 22 mars 2019 18:47:19 GMT+01:00, Joshua Marshall
><j.marshall@arroyo.io>
>> a écrit :
>>>
>>> Hello all,
>>>
>>> I was told in IRC to post my possibly unreasonable feature requests
>here.
>>>
>>> I've been thinking more on what Guix might be able to do, and what
>would
>>> make it more useful for a few of my past jobs.  I'd like to see it
>take on
>>> the ability to have a per-installation target cgroup, network
>namespace,
>>> and filesystem chroot settings set with defaults which are
>overridable at
>>> invocation.  In this way, a user could install and use packages with
>>> mutually incompatible dependencies (I talked about this with a few
>people
>>> on IRC) like what happens with python.  If this kind of
>functionality were
>>> added, it would largely supplant Docker, virtualenv, pip, poetry,
>apk,
>>> pacman, and probably a few other tools at my company which are there
>just
>>> to handle this kind of frailness.  From this, I could also see an
>entry
>>> point to adding build module support to start to replace tools like
>Make,
>>> CMake, Meson, Bazel, and so on.
>>>
>>> These expand the scope of Guix quite a bit, but I think these are
>needed
>>> for it to really feel logically complete.  Does all this make sense?
>>>
>>>
>>> Please be advised that this email may contain confidential
>information.
>>> If you are not the intended recipient, please notify us by email by
>>> replying to the sender and delete this message. The sender disclaims
>that
>>> the content of this email constitutes an offer to enter into, or the
>>> acceptance of, any agreement; provided that the foregoing does not
>>> invalidate the binding effect of any digital or other electronic
>>> reproduction of a manual signature that is included in any
>attachment.
>>>
>>> <https://twitter.com/arroyo_networks>
>>> <https://www.linkedin.com/company/arroyo-networks>
>>> <https://www.github.com/ArroyoNetworks>
>>>
>>
>> --
>> Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez
>excuser ma
>> brièveté.
>>
>
>-- 
>
>
>
>Please be advised that this email may contain confidential information.
>
>If you are not the intended recipient, please notify us by email by 
>replying to the sender and delete this message. The sender disclaims
>that 
>the content of this email constitutes an offer to enter into, or the 
>acceptance of, any agreement; provided that the foregoing does not 
>invalidate the binding effect of any digital or other electronic 
>reproduction of a manual signature that is included in any attachment.
>
>
> 
><https://twitter.com/arroyo_networks>   
><https://www.linkedin.com/company/arroyo-networks>   
><https://www.github.com/ArroyoNetworks>

-- 
Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma brièveté.

[-- Attachment #2: Type: text/html, Size: 9594 bytes --]

Re: Feature requests

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 3059 bytes --]

Hi Joshua,

Joshua Marshall <j.marshall@arroyo.io> writes:

[...]

> I'd like to see it take on
> the ability to have a per-installation target cgroup, network namespace,
> and filesystem chroot settings set with defaults which are overridable at
> invocation.

me too and the only missing point above (AFAIU) is network isolation for
Guix containers, I mean one created via `guix environment` or `guix
system container`)

having that, the "last mile" in *obsoleting* tools like Docker &
Co. (e.g. kubernetes, even openstack probably) is to have a declarative
way to setup containers, something like `containers.<name>` from NixOS
[1]

...and a set of Guix services to declaratively `scale out` an
infrastrtucture: a layer 4+7 proxy (e.g. haproxy, missing in Guix),
Software Defined Network (openvswitch, got it!), Software Defined
Storage (ceph: we have the pachage but missing the service AFAIU)

anyway: containers are here to solve infrastructural problems, not
development environments problems :-)

[1] https://nixos.org/nixos/manual/index.html#sec-declarative-containers

> In this way, a user could install and use packages with
> mutually incompatible dependencies (I talked about this with a few people
> on IRC) like what happens with python.  If this kind of functionality were
> added, it would largely supplant Docker,

you cited Docker so I guess you are using containers as a mean to
isolate *development environment* each other and from the *production
environment*, not to build an insfrastructure of isolated set of
processes (including networking layer) - let's call them nodes -
possibly distributed on several hosts

in this thread Julien already explained how to achieve this with `guix
environment`: with Guix (and Nix, the *only* other sofware natively
permitting this) you don't need to install a container to have
*isolated* development environments

AFAIU in *many*, many, many use cases containers (Docker, LXC and so on)
are _not_ used as an infrastructural component but as a development
tool: Guix obsoletes this thanks to its native isolated environments
(made possible by The Store)

I hope more and more developers will realize this since this is
_for_sure_ a big win for the entire free software community (no more
python virtualenv clones, *please*)

> virtualenv, pip, poetry, apk,
> pacman, and probably a few other tools at my company which are there just
> to handle this kind of frailness.

`guix environment` and the package definition programming interface [2]
(it's really easy to learn, believe me :-) ) are your best friends here

you can even `guix pack` sofware bundles (e.g. in Docker format) and
distribute it to your internal/external customers who are still not able
to use Guix to install them

[2] https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packages.html#Defining-Packages

[...]

HTH to better explain how development works in a Guix environment :-)

Gio

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Feature requests

[Joshua Marshall]

[-- Attachment #1: Type: text/plain, Size: 4043 bytes --]

Thank you!  I have ~500 pages of other stuff to read this week, but I'll
get to this as soon as I can.

On Mon, Mar 25, 2019 at 5:40 AM Giovanni Biscuolo <g@xelera.eu> wrote:

> Hi Joshua,
>
> Joshua Marshall <j.marshall@arroyo.io> writes:
>
> [...]
>
> > I'd like to see it take on
> > the ability to have a per-installation target cgroup, network namespace,
> > and filesystem chroot settings set with defaults which are overridable at
> > invocation.
>
> me too and the only missing point above (AFAIU) is network isolation for
> Guix containers, I mean one created via `guix environment` or `guix
> system container`)
>
> having that, the "last mile" in *obsoleting* tools like Docker &
> Co. (e.g. kubernetes, even openstack probably) is to have a declarative
> way to setup containers, something like `containers.<name>` from NixOS
> [1]
>
> ...and a set of Guix services to declaratively `scale out` an
> infrastrtucture: a layer 4+7 proxy (e.g. haproxy, missing in Guix),
> Software Defined Network (openvswitch, got it!), Software Defined
> Storage (ceph: we have the pachage but missing the service AFAIU)
>
> anyway: containers are here to solve infrastructural problems, not
> development environments problems :-)
>
> [1] https://nixos.org/nixos/manual/index.html#sec-declarative-containers
>
> > In this way, a user could install and use packages with
> > mutually incompatible dependencies (I talked about this with a few people
> > on IRC) like what happens with python.  If this kind of functionality
> were
> > added, it would largely supplant Docker,
>
> you cited Docker so I guess you are using containers as a mean to
> isolate *development environment* each other and from the *production
> environment*, not to build an insfrastructure of isolated set of
> processes (including networking layer) - let's call them nodes -
> possibly distributed on several hosts
>
> in this thread Julien already explained how to achieve this with `guix
> environment`: with Guix (and Nix, the *only* other sofware natively
> permitting this) you don't need to install a container to have
> *isolated* development environments
>
> AFAIU in *many*, many, many use cases containers (Docker, LXC and so on)
> are _not_ used as an infrastructural component but as a development
> tool: Guix obsoletes this thanks to its native isolated environments
> (made possible by The Store)
>
> I hope more and more developers will realize this since this is
> _for_sure_ a big win for the entire free software community (no more
> python virtualenv clones, *please*)
>
> > virtualenv, pip, poetry, apk,
> > pacman, and probably a few other tools at my company which are there just
> > to handle this kind of frailness.
>
> `guix environment` and the package definition programming interface [2]
> (it's really easy to learn, believe me :-) ) are your best friends here
>
> you can even `guix pack` sofware bundles (e.g. in Docker format) and
> distribute it to your internal/external customers who are still not able
> to use Guix to install them
>
> [2]
> https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packages.html#Defining-Packages
>
> [...]
>
> HTH to better explain how development works in a Guix environment :-)
>
> Gio
>
> --
> Giovanni Biscuolo
>
> Xelera IT Infrastructures
>

-- 



Please be advised that this email may contain confidential information. 
If you are not the intended recipient, please notify us by email by 
replying to the sender and delete this message. The sender disclaims that 
the content of this email constitutes an offer to enter into, or the 
acceptance of, any agreement; provided that the foregoing does not 
invalidate the binding effect of any digital or other electronic 
reproduction of a manual signature that is included in any attachment.


 
<https://twitter.com/arroyo_networks>   
<https://www.linkedin.com/company/arroyo-networks>   
<https://www.github.com/ArroyoNetworks>

[-- Attachment #2: Type: text/html, Size: 5814 bytes --]

Declarative containers

[Ludovic Courtès]

Hello,

Giovanni Biscuolo <g@xelera.eu> skribis:

> having that, the "last mile" in *obsoleting* tools like Docker &
> Co. (e.g. kubernetes, even openstack probably) is to have a declarative
> way to setup containers, something like `containers.<name>` from NixOS
> [1]

[...]

> [1] https://nixos.org/nixos/manual/index.html#sec-declarative-containers

Given that we already have ‘guix system container’, declarative
containers are a low hanging fruit.  We could have a ‘container’ (or
‘sub-system’?) service; you’d pass it an <operating-system> and it’d
create a Shepherd service that runs that OS in a container.

That would be fun!

Ludo’.

Re: Declarative containers

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 1155 bytes --]

Hello,

thanks for your feedback Ludo'! :-)

Ludovic Courtès <ludo@gnu.org> writes:

[...]

> Given that we already have ‘guix system container’, declarative
> containers are a low hanging fruit.

"low hanging fruit" is much more expressive than "last mile" :-)

> We could have a ‘container’ (or ‘sub-system’?) service;

mumble: `container` is so abused that it's starting to become a buzzword
to my ears, `sub-system` is probably more semantic

> you’d pass it an <operating-system> and it’d create a Shepherd service
> that runs that OS in a container.

what is the method used to instantiate containers with Guix?
I know lxc/lxd and systemd-nspawn but can't still understand what's used
here

> That would be fun!

yep, a *lot* of fun

also, given Guile homoiconicity a web based (who wants web based
interfaces?!? bah?) and/or declarative orchestration system built on top
of Guix+GWL would be the next (not so low) hanging fruit :-D

uh, if only I was able to hack in guile... I'll get there but I'm so
sorry I cannot help _now_ :-S

Gio'.

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Declarative containers

[Ludovic Courtès]

Hi,

Giovanni Biscuolo <g@xelera.eu> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> We could have a ‘container’ (or ‘sub-system’?) service;
>
> mumble: `container` is so abused that it's starting to become a buzzword
> to my ears, `sub-system` is probably more semantic

The Hurd has had this thing called “sub-Hurd”, which is similar to what
we’re talking about.

>> you’d pass it an <operating-system> and it’d create a Shepherd service
>> that runs that OS in a container.
>
> what is the method used to instantiate containers with Guix?

‘guix system container’ creates an executable that starts the container.
That executable is a Guile program that uses (gnu build
linux-container), a module that provides facilities to create processes
under separate name spaces, etc.

IOW all the functionality is provided by Guix; no systemd-nspawn,
bubblewrap, lxc, etc.

Ludo’.