Thoughts on making Guix even better

Thoughts on making Guix even better

[Raghav Gururajan]

Hello Guix!

I have been thinking about this for a long time and would like to share it now.

The transactional upgrades and roll-backs are available to both Guix Package and Guix System. But I see a important difference which might be crucial to guix's development and use.

GUIX PACKAGE:

The guix package transactions are MODULAR. That is, you can upgrade packages selectively. For example, you can upgrade all packages except one/few (or) only upgrade one/few.

GUIX SYSTEM:

The guix system transactions are NON-MODULAR. That is, you cannot selectively reconfigure certain parts of the system. For example, you either reconfigure the system as a whole (or) you do not reconfigure the system at all.

IMPLICATIONS:

Lets assume we have 5 packages in profile. Package 1, 3 and 5 has non-critical updates. Package 4 has non-critical update but it breaks. Package 2 has critical update (CVE). We can either upgrade all packages except package 4 (or) we can upgrade only package 2.

Lets assume we have 5 services/packages in system. Package/Service 1, 3 and 5 has non-critical updates. Package/Service 4 has non-critical update but it breaks. Package/Service 2 has critical update (CVE). Now, when we reconfigure the system, all packages/services will upgrade, package/service 4 will break the system. We can of course do '--roll-back' and take the system to previous working state. But that will leave the system with critical vulnerability. Therefore, we cannot reconfigure package/service 2 or any other parts of the system, until the package/service 4 is fixed. This window/gap puts guix system at great risk and instability.

SUGGESTION:

We can brain-storm and implement a way to make guix system transactions modular. Any ideas?

Thank you!

Regards,
Raghav "RG" Gururajan.

Thoughts on making Guix even better

[Leo Prikler]

Hello Raghav!
> Lets assume we have 5 packages in profile. Package 1, 3 and 5 has
> non-critical 
> updates. Package 4 has non-critical update but it breaks. Package 2
> has 
> critical update (CVE). We can either upgrade all packages except
> package 4 (or) 
> we can upgrade only package 2.
> 
> Lets assume we have 5 services/packages in system. Package/Service 1,
> 3 and 5 
> has non-critical updates. Package/Service 4 has non-critical update
> but it 
> breaks. Package/Service 2 has critical update (CVE). Now, when we
> reconfigure 
> the system, all packages/services will upgrade, package/service 4
> will break 
> the system. We can of course do '--roll-back' and take the system to
> previous 
> working state. But that will leave the system with critical
> vulnerability. 
> Therefore, we cannot reconfigure package/service 2 or any other parts
> of the 
> system, until the package/service 4 is fixed. This window/gap puts
> guix system 
> at great risk and instability.
This is not as much a guix package vs. guix system issue as it is an
issue of explicit manifests against implicit ones.  If you use guix
package with manifests and without inferiors, you will have the same
problem.  Likewise, you can use inferiors in your config.scm to
mitigate some of those issues.  At least it works for the kernel, but
it should in theory also work for packages.

The problem with inferiors as a solution to this problem is, that it
doesn't address the issues of services.  You'd have to use the current
service structure with an inferior-package, which is not always what
you want, specifically when the introduction of a new field to that
service causes an issue.  In addition to that, finding all package
references and patching them to not include some breaking package (say
e.g. the newest mesa version, which depending on your graphics card may
or may not cause issues) can be very tedious depending on what is
referenced where.  Perhaps a lookup-inferior-services procedure might
help here.  

Overall, there are also some "not so fun" things when dealing with
inferior packages.  For one (car (lookup-inferior-packages ...)) is
quite a mouthful, especially when you know you'll always want the first
result or there is only one to begin with.  I'd welcome a procedure to
turn an inferior into a procedure that always returns the first match. 
IIRC inferior packages are also not always accepted as packages, but
I'd welcome being proven wrong about that.

You can also modularize guix system by wrapping each and every service
in a module which you either re-export from guix proper or -- in case
of some failure -- implement on your own.  That's a lot of work
however.

TL;DR: You can "modularize" transactions with 'guix system' in the same
way you modularize 'guix package -m' (the "-m" means "not modular" ;P).
 
Regards,
Leo

PS: What you're envisioning is probably a front-end, that obscures the
very existence of a config.scm by managing one that is just as verbose
as guix-generated manifests are.  However, this is not really a
solution as it fails to address the need for a (human-readable) initial
configuration.  The interface would also be a pain to deal with as each
service comes with its own configuration record allowing arbitrary lisp
expressions that one would have to write on the command line.

Re: Thoughts on making Guix even better

[Jonathan Frederickson]

On Feb 22, 2020, at 9:49 PM, Raghav Gururajan <raghavgururajan@disroot.org> wrote:
> 
> The guix system transactions are NON-MODULAR. That is, you cannot selectively reconfigure certain parts of the system. For example, you either reconfigure the system as a whole (or) you do not reconfigure the system at all.

This is something that also made it a bit difficult to get started writing services: you can’t work on a service in isolation and enable it on your current system by itself, which would be nice. I ended up building an operating-system container with just that service running, which works, but increases the time taken for each code-test cycle.

Re: Thoughts on making Guix even better

[Raghav Gururajan]

Hello Leo!

> This is not as much a guix package vs. guix system issue as it is an
> issue of explicit manifests against implicit ones. If you use guix
> package with manifests and without inferiors, you will have the same
> problem. Likewise, you can use inferiors in your config.scm to
> mitigate some of those issues. At least it works for the kernel, but
> it should in theory also work for packages.

I see.

> PS: What you're envisioning is probably a front-end, that obscures the
> very existence of a config.scm by managing one that is just as verbose
> as guix-generated manifests are. However, this is not really a
> solution as it fails to address the need for a (human-readable) initial
> configuration. The interface would also be a pain to deal with as each
> service comes with its own configuration record allowing arbitrary lisp
> expressions that one would have to write on the command line.

I think we can still maintain the guix way of doing config.scm and also bring modularity. My thought is, what if we could split the operating-system procedures into smaller procedures, such as, kernel, system-wide packages, services etc. into separate procedures? So if a user passes the procedure name to the `guix system reconfigure` command, then only that procedure is reconfigured. For example, we can reconfigure kernel of the system without reconfiguring packages and services.

What do you think?

Regards,
RG.

Re: Thoughts on making Guix even better

[Ludovic Courtès]

Hi,

"Raghav Gururajan" <raghavgururajan@disroot.org> skribis:

> The guix system transactions are NON-MODULAR. That is, you cannot selectively reconfigure certain parts of the system. For example, you either reconfigure the system as a whole (or) you do not reconfigure the system at all.
>
> IMPLICATIONS:
>
> Lets assume we have 5 packages in profile. Package 1, 3 and 5 has non-critical updates. Package 4 has non-critical update but it breaks. Package 2 has critical update (CVE). We can either upgrade all packages except package 4 (or) we can upgrade only package 2.
>
> Lets assume we have 5 services/packages in system. Package/Service 1, 3 and 5 has non-critical updates. Package/Service 4 has non-critical update but it breaks. Package/Service 2 has critical update (CVE). Now, when we reconfigure the system, all packages/services will upgrade, package/service 4 will break the system. We can of course do '--roll-back' and take the system to previous working state. But that will leave the system with critical vulnerability. Therefore, we cannot reconfigure package/service 2 or any other parts of the system, until the package/service 4 is fixed. This window/gap puts guix system at great risk and instability.

On one hand, I agree that it’d be nice to be able to update just parts
of the system, like you explain.

On the other hand, that would lead to an unknown and possibly
unreproducible system state, which defeats what declarative
(“non-modular”) system upgrades bring.

Besides, I don’t see how one could introduce this “imperative” approach
at the system level, technically.

All in all, it would be best if the situations that make “modular system
upgrades” appear necessary didn’t occur in the first place.

Thoughts?

Ludo’.

Re: Thoughts on making Guix even better

[Gábor Boskovits]

[-- Attachment #1: Type: text/plain, Size: 3216 bytes --]

Hello,

Ludovic Courtès <ludo@gnu.org> ezt írta (időpont: 2020. márc. 8., Vas
21:54):

> Hi,
>
> "Raghav Gururajan" <raghavgururajan@disroot.org> skribis:
>
> > The guix system transactions are NON-MODULAR. That is, you cannot
> selectively reconfigure certain parts of the system. For example, you
> either reconfigure the system as a whole (or) you do not reconfigure the
> system at all.
> >
> > IMPLICATIONS:
> >
> > Lets assume we have 5 packages in profile. Package 1, 3 and 5 has
> non-critical updates. Package 4 has non-critical update but it breaks.
> Package 2 has critical update (CVE). We can either upgrade all packages
> except package 4 (or) we can upgrade only package 2.
> >
> > Lets assume we have 5 services/packages in system. Package/Service 1, 3
> and 5 has non-critical updates. Package/Service 4 has non-critical update
> but it breaks. Package/Service 2 has critical update (CVE). Now, when we
> reconfigure the system, all packages/services will upgrade, package/service
> 4 will break the system. We can of course do '--roll-back' and take the
> system to previous working state. But that will leave the system with
> critical vulnerability. Therefore, we cannot reconfigure package/service 2
> or any other parts of the system, until the package/service 4 is fixed.
> This window/gap puts guix system at great risk and instability.
>
> On one hand, I agree that it’d be nice to be able to update just parts
> of the system, like you explain.
>
> On the other hand, that would lead to an unknown and possibly
> unreproducible system state, which defeats what declarative
> (“non-modular”) system upgrades bring.
>
> Besides, I don’t see how one could introduce this “imperative” approach
> at the system level, technically.
>
> All in all, it would be best if the situations that make “modular system
> upgrades” appear necessary didn’t occur in the first place.
>
> Thoughts?
>

I believe that there are two points where it would be possible to improve
the situation.
1. Improve tooling to modularize the  configurations: like allowing an
inferior like feature for services, and adding tests to this (this is a way
of service versioning), or even setting up a convention to include scheme
files from a location, like ./services.d files get included, and the
expression they evaluated to are added to the services field if something
like this makes sense.
Make it possible for services to specify upgrade actions to run when the
version changes, or to fail when manual intervention is needed for a
correct upgrade.
2. Allow post install action configuration, for example stating that this
list of services should be restarted. Also allow to guess the right post
install action if none specified, and allow the services to add features to
this guessing mechanism, like which configuration changes require restart.
Make it possible to reload services by arranging their configs in a way
that reloads work.

In both of these cases it might be needed to inspect the previous system,
but the system provision information should be enough for that. Wdyt?

>
> Ludo’.
>
Best regards,
g_bor

>
>

[-- Attachment #2: Type: text/html, Size: 4130 bytes --]

Re: Thoughts on making Guix even better

[Konrad Hinsen]

Hi everyone,

>      > The guix system transactions are NON-MODULAR. That is, you cannot
>     selectively reconfigure certain parts of the system. For example,
>     you either reconfigure the system as a whole (or) you do not
>     reconfigure the system at all.

Today's software systems are inherently non-modular. Guix is the most 
advanced technology we have to work around this problem, but it doesn't 
solve it. The problem is so hard to solve because it requires most of 
all a change in everyone's attitude towards software development, and 
thus very probably different organizational structures (because of 
Conway's Law).

Cheers,
   Konrad.