From mboxrd@z Thu Jan 1 00:00:00 1970 From: Giovanni Biscuolo Subject: Re: Thoughts on stateful services in Guix Date: Wed, 05 Feb 2020 14:03:08 +0100 Message-ID: <87r1z9chkj.fsf@roquette.mug.biscuolo.net> References: <87r1zexcu3.fsf@gmail.com> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:50264) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1izKLY-0000je-S5 for guix-devel@gnu.org; Wed, 05 Feb 2020 08:04:02 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1izKLV-0000PD-9y for guix-devel@gnu.org; Wed, 05 Feb 2020 08:04:00 -0500 Received: from ns13.heimat.it ([46.4.214.66]:40956) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1izKLU-0000Dm-VH for guix-devel@gnu.org; Wed, 05 Feb 2020 08:03:57 -0500 In-Reply-To: <87r1zexcu3.fsf@gmail.com> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane-mx.org@gnu.org Sender: "Guix-devel" To: alex.sassmannshausen@gmail.com, Guix-devel --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Alex, I'm very interested in deploying "as stateless as possible" services with Guix, thank you for sharing your thougts on this! I'm still non able to contribute code fot this task... I'm working on it Alex Sassmannshausen writes: > As a result of FOSDEM conversations today I felt inspired to put some > thoughts on paper about an area where I think we currently run into > complications. I also think that deploying services in a functional way (ala Guix packages to be clear) is _complicated_ and... it's a problem to be clear: this complication it's *absolutely not* Guix specific, but it's something that becomes very clear when you want to use the declarative approach to "everithing" once you've got Guix :-O (and after you were deluded by the promises from tools like Puppet and alike) > Not sure if it is appropriate to write this blog-post-ish contribution > here, but as I don't have a blog, and it's about Guix development, I > figured it might be OK. maybe with a little help by someone more qualified than me this should become a Guix blog post... or better be generalized ("how to write staletess services") and included in the cookbook anyway, I think your analisys is very helpful in discussing how Guix can help fighting the "state dragon" I've some comment and questions > Best wishes, > > Alex > > 1 Introduction > =E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90= =E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90 [...] > Disciplined developers can implement many services in such a way that > their statefulness is delegated to other dedicated services. In this > way the problem of statefulness can be isolated. However, many > existing useful services /have not/ been implemented in such a way. do you mean upstream? [...] from now on you used Drupal as an example: I do not know how Drupal works, but I know other problematic web services :-) > 2.3.1 Enter the state dragon > =E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C= =E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2= =95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95= =8C=E2=95=8C=E2=95=8C=E2=95=8C > > But hang-on=E2=80=A6 if Mysql service is a dependency, and we store sta= te in a > Mysql DB, in this regard some time ago I found http://gfxmonk.net/2015/01/03/nixos-and-stateless-deployment.html (see "Lumps of impurity") a good explanation on what should be state and what not; this is the relevant part: =2D-8<---------------cut here---------------start------------->8--- NixOS keeps the =E2=80=9COS=E2=80=9D parts of the machine stateless, meanin= g that the only state you need to manage is that which you create yourself. From the top of my head (not an exhaustive list): =2D /var, /tmp, /run: stateful, unmanaged =2D each user=E2=80=99s $HOME directory: stateful, unmanaged (rarely used o= n a server, though) =2D Users & Groups: stateless =2D Installed software: stateless =2D Installed services (using systemd): stateless =2D All program configuration (i.e all of /etc): stateless =2D Kernel & kernel modules, grub configuration: stateless (but requires a = reboot to activate) =2D Disk mounts: stateless By =E2=80=9Cstateless=E2=80=9D, I mean that the given item is generated ent= irely from the pure nix expression of the system, and isn=E2=80=99t affected by any previous state. =2D-8<---------------cut here---------------end--------------->8--- I was not able to find a similar description in some Guix document... but it's OT now :-) (...and software config stored in $HOME should be stateless, also; guix-home-manager [1] is a _very_ interesting work on this topic) storing state it's obviously the job of many services, what's bad it's some software (Drupal? Discourse for sure, Wordpress for sure) store (part of?) configuration as status; some software provides a CLI tool (usually as a afterthougt, e.g. wp-cli) to imperatively configure it, but it's always treated as status in other words: some (usually web) services "embeds" config with status and provide one or more interface to configure, others clearly separates config from status leaving the "burden" of writing configuration (usually in /etc) to sysadmins... and that software is "easy" to integrate as a stateless (and functional) Guix service my question is: can we work around this very bad design choiche? can we patch the package and have stateless config for, e.g., Drupal? I see this someway similar to "de-blobbing" our packages: how hard it's to do depends on upstream... unfortunately not all developers agrees that stateful config is a problem (in a similar way not all agrees that distributing binary-blobs it's a problem) > what happens when we upgrade to a newer version of Drupal, > with a different schema? Would this service instantiation change the > state of our system? Could we still roll-back? oh yes, schema migration of status is also a problem to manage when we upgrade software... and usually we do it "by hand" (making backups of status, ecc.) schema migration is usual when status is stored in a database, theoretically schema migration could be applyed also when storing state in XML (YAML?) in files, but I've never seen this use case your proposal (below) is interesting! > And remember that broken symlink we introduced in the package? That > files/ subdirectory contains /at least/ a stateful configuration > file and that's the main problem: configuration as status > =E2=80=94 but also Drupal modules, themes, modules and themes should be packaged separately, IMHO (and yes, they sometimes can be installed via the web interface... and that's a problem) the general problem is that some services have become a software blob providing all sort of features: package installation, configuration editing, what else? > uploaded files and caches. uploaded files are status and generally not a problem, no? caches are status and sometimes a problem and they need to be wiped since not all software is "obsolete-cache-resilient"; however it's not still clear how to manage this kind of problem Guix-wide, I've read several problems was resolved in the past wiping cache (e.g. with gnome) but AFAIU there is no general consensus on a general automated method anyway I think it's not a good idea to backup caches as part of the state-dump/state-load procedure you propose below > What > happens when we upgrade Drupal? Can we guarantee they won't be > irrevocably changed as part of the upgrade? What happens when we > roll-back? disaster :-O (even with state, in case of schema migrations) [...] > 2.3.2 Fakin' it til we're makin' it > =E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C= =E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2= =95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95= =8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C=E2=95=8C= =E2=95=8C=E2=95=8C > > Can we get those guarantees back? I fear not. AFAIU there is no point in defining services in Guix if we have no guarantees on the main promise of Guix: no? ...or put in another way: is there any defined Guix service with a stateful (non declarative) configuration file? what I mean is: we still do not have services like Drupal, or Discourse or VTiger _because_ they have a stateful configuration system ad it is not easy to "workaround" this foundamental pitfall (in some case it's also not easy to package the software) [...] > Working backwards, to ensure we can roll-back we need to make sure > that we have access to the state at that point in history that we want > to roll-back to. When we roll-back from version 2 of Drupal to > version 1, we must make sure we still have the state of version 1. this approach is *very* interesting to allow rolling-back in the event of a failed (usually automated) migration of the state (e.g. for a schema change): thanks! but using this approach also for config would mean we lose the declarative approach for (some of) our services, and I'm not sure it's a viable long term solution; please consider that declarative configuration means that one day (it's not an "if") Guix users will be able to configure an entire infrastructure via one (we have Emacs today) or more UI, one of them will be GUI and/or HTML5 [...] > 2.4 The stateful service shimmy > =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80 > > Here's the proposed general shim then: > > 1) We define a new type of shepherd service, a /stateful service/, > which has additional actions: state-dump-shim and > state-load-shim. state-dump and state-load could be useful for stateless services, too [...] > What does this give us? > > When a migration to a new service revision causes a deployed bit of > software to stop working as expected, we can do a `guix system > =E2=80=93roll-back`. This will then also restore the service's state t= o the > previous version and thus *should* restore operations. this would be a *great* piece of automation for all of us, since it's good practice to "manually" back up state before migrations, so rolling-back to a previous schema is possible in case of failures (I've had some of that in my experience); as I said, this approach is _very_ useful even for stateless Guix services [...] > 3 A generalisation: the Stateful-Service Service > =E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90= =E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2= =95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95= =90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90= =E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2= =95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90=E2=95=90 > > State dumping and restoration *should* be generalisable. > > It should normally consist of one or more operations of: > =E2=80=A2 running a special program to dump data, encrypting the dump, = and > storing it, together with the service revision hash, in a known > location. > =E2=80=A2 tarring up a directory tree, encrypting the archive and stori= ng it, > together wtih the service revision hash, in a known location. > > These operations should be able to be provided by a daemon, managed by > shepherd, which can be configured with a gpg key for encryption, a > mechanism for decryption (interactive or programmatic), a location for > storing/retrieving dumps (local or remote?), and a DSL for mapping > data dump / data restoration program invocations or file-system > locations to data dumps it already knows of. very interesting, I'd wish I'd be able to provide a POC! :-S thanks a lot! Giovanni [1] https://framagit.org/tyreunom/guix-home-manager =2D-=20 Giovanni Biscuolo Xelera IT Infrastructures --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEERcxjuFJYydVfNLI5030Op87MORIFAl46vQ0ACgkQ030Op87M ORIKSBAAo3CptxixlqfMp25RfNnxzusNBbG98Fay5wEM7FdqCltBnXrgCENBoLkB WfE0sHghoCLQowvELcXwPYyq/OCrMUunbr7YH+79CL6MsN4ilifYIM0pJeFfVwsG xfkWkV27vpoU5pNufGfDmyePjDw6FxrE5f/x/zOb+J0k8KFxc+AJblKR1q++u3f0 pN9W/0c403reojhkz3r+Yb7X5qo7xU4zadVlArqUXqVBEg0fyMWbuCPoZQsh11ej ERZvdOzD79HpEHQgyK7WDvISGlIJmPiFVr2VXwB5EegRT8l4ic3oMWBBXCh1mnKP 0OF5LlcNPHRCoch31mCQgbenkrfTtIELZRUx6O4I2As2gopV/Xq5M3WFuEObtt5j Vn9ehAQv+7zW1i1bYPdnBWEzJ5N5R4F4WQQcL+5ZJTYRTGHuf/sy2Xoc2TvabwjT jefUansagHJyHMgQM3dAQmo3GDEddskOsByJjo0EbRypxSYv7/0KHHfSS4MG2sJ2 iRyvmGmdf/SWD3InS+UocXisJbnEWa8bijD6zPg6vqr1tZNAYXK8d4WqjD4HIqpe PpE+rvXEFUaiXPuH/ui9EvvHu1PMqBCtboN03e9K0YH08+QKQRuzKiXO0IDpOKeT NA6lgnDh8EphIdYWEpuQUO8P3b3DKotV7gX19/KI9iBovDp1vaE= =7S+s -----END PGP SIGNATURE----- --=-=-=--