Stateful system directories

Stateful system directories

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1712 bytes --]

On Fri, Oct 18, 2019 at 05:08:20AM +0200, Ricardo Wurmus wrote:
> 
> Kei Kebreau <kkebreau@posteo.net> writes:
> 
> > Ricardo Wurmus <rekado@elephly.net> writes:
> >
> >> Kei Kebreau <kkebreau@posteo.net> writes:
> >>
<snip>
> >>
> >> Have you tried removing /var/lib/gdm and the contents of your user
> >> account’s .local/share/gnome* directories?
> >
<snip>
> 
> ~/.local/share/gnome-shell/application_state is a common problem.  It
> contains some state that different versions of GNOME seem to be choking
> on.  There are some other files like ~/.cache/gnome* that might affect
> GNOME and prevent starting after upgrades.  It’s frustrating.
> 
> /var/lib/gdm is the home directory of the gdm account, and it too can
> accumulate state.  In my opinion /var/lib/gdm should always be recreated
> on every boot.
> 

Ignoring the directories in users' home directories, /var/lib/gdm has
been a source of pain on GNOME upgrades, and we still have some problems
with /var/cache/fontconfig and I believe there is something else with
permissions if you switch between ntp and openntpd. I actually have the
following snippet in my OS-config:

;; This directory shouldn't exist
(file-system
  (device "none")
  (mount-point "/var/cache/fontconfig")
  (type "tmpfs")
  (flags '(read-only))
  (check? #f))

While we work on fixing these does it make sense to modify some of these
services to unconditionally recreate their home directories on
boot/activation?

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Stateful system directories

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 3108 bytes --]

Hi Efraim,

Efraim Flashner <efraim@flashner.co.il> writes:

[...]

> I actually have the following snippet in my OS-config:
>
> ;; This directory shouldn't exist
> (file-system
>   (device "none")
>   (mount-point "/var/cache/fontconfig")
>   (type "tmpfs")
>   (flags '(read-only))
>   (check? #f))

yes! now that I read this I see how to workaround this class of
problems, thanks! 

This snippet IMHO deserves a section in the Cookbook: would you please
like to expand this in a proper section/subsection and submit a patch?

> While we work on fixing these

we in Guix you mean? shouldn't it be fixed upstream?

obviously we should help upstream as much as possible giving them
feedback; I don't know about fontconfig, but AFAIU Gnome upstream is
well aware (do I miss something?) of the problems with "statefulness"
compromising starting critical piece of system software like GDM and
*sysadmins* having to work that around deleting "system level"
directories and *each user* deleting one or more of their
.local/<something> files

how can Guix work on fixing this if not by providing workarounds in
system or user services (when will be available) that makes it easier to
sysadmins and users to fix that problems at a different level of their
systems?

to expand my reasoning just to give a little bit of context, I give this
other example: I use Nextcloud on NixOS (still not in Guix), it turns
out that there are a couple of issues that makes practically (I mean in
a practical way) impossible (AFAIU) using Nextcloud in a stateless way,
and there is no even a workaround; details here:
https://github.com/NixOS/nixpkgs/issues/49783#issuecomment-481350460
"[...] the underlying bug is a missing distinction between configuration
and state upstream…"

AFAIU this falls in the same class of problems as the Gnome one
described in the thread you are referencing to (also fontconfig?) 

so, in my POV, one of the interesting "side effects" (or probably main
effect? :-) ) of *using* distributions like Guix System or NixOS - which
promotes stateless configuration from the packages build phase up to
services provisioning [1] - is that users will be affected by this class
of problems while others simply ignores them because they prefer to use
an imperative way to manage their systems... and upstream does not get
enough pressure from its userbase to fix this class of problems

in my case, the Nextcloud problem described above is also refraining me
from investing some resources in trying to package it for Guix (also as
a service, obviously)

> does it make sense to modify some of these services to unconditionally
> recreate their home directories on boot/activation?

IMHO yes, it makes sense; unfortunately in cases like that described
above for Nextcloud, this could have destroying effects on the service,
so we cannot workaround :-S

Thanks! Gio'

[...]


[1] differences in approach to "stateless purity" apart, but they are
secondary in this context, IMHO

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Stateful system directories

[P]

Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, October 18, 2019 10:05 AM, Giovanni Biscuolo <g@xelera.eu> wrote:

> "[...] the underlying bug is a missing distinction between configuration
> and state upstream…"

See also the numerous programs that treat ~/.config like it was also cache and state.

Re: Stateful system directories

[Ricardo Wurmus]

Efraim Flashner <efraim@flashner.co.il> writes:

> On Fri, Oct 18, 2019 at 05:08:20AM +0200, Ricardo Wurmus wrote:
>>
>> Kei Kebreau <kkebreau@posteo.net> writes:
>>
>> > Ricardo Wurmus <rekado@elephly.net> writes:
>> >
>> >> Kei Kebreau <kkebreau@posteo.net> writes:
>> >>
> <snip>
>> >>
>> >> Have you tried removing /var/lib/gdm and the contents of your user
>> >> account’s .local/share/gnome* directories?
>> >
> <snip>
>>
>> ~/.local/share/gnome-shell/application_state is a common problem.  It
>> contains some state that different versions of GNOME seem to be choking
>> on.  There are some other files like ~/.cache/gnome* that might affect
>> GNOME and prevent starting after upgrades.  It’s frustrating.
>>
>> /var/lib/gdm is the home directory of the gdm account, and it too can
>> accumulate state.  In my opinion /var/lib/gdm should always be recreated
>> on every boot.
>>
>
> Ignoring the directories in users' home directories, /var/lib/gdm has
> been a source of pain on GNOME upgrades, and we still have some problems
> with /var/cache/fontconfig and I believe there is something else with
> permissions if you switch between ntp and openntpd. I actually have the
> following snippet in my OS-config:
>
> ;; This directory shouldn't exist
> (file-system
>   (device "none")
>   (mount-point "/var/cache/fontconfig")
>   (type "tmpfs")
>   (flags '(read-only))
>   (check? #f))

Ah, neat.  A bit heavy-handed, of course, but neat :)

> While we work on fixing these does it make sense to modify some of these
> services to unconditionally recreate their home directories on
> boot/activation?

I think there’s no compelling reason to keep /var/lib/gdm state across
reboots.  When this goes wrong it’s very painful, and that’s much more
significant than any savings (e.g. in startup times) that it might bring
us.  So here’s my vote for letting the gdm service recreate its home
directory unconditionally, perhaps with a toggle to disable this
behaviour (e.g. when someone wants to use a different directory and
somehow alter GDM behaviour this way).

--
Ricardo

Re: Stateful system directories

[Ricardo Wurmus]

Giovanni Biscuolo <g@xelera.eu> writes:

> Hi Efraim,
>
> Efraim Flashner <efraim@flashner.co.il> writes:
>
> [...]
>
>> I actually have the following snippet in my OS-config:
>>
>> ;; This directory shouldn't exist
>> (file-system
>>   (device "none")
>>   (mount-point "/var/cache/fontconfig")
>>   (type "tmpfs")
>>   (flags '(read-only))
>>   (check? #f))
>
> yes! now that I read this I see how to workaround this class of
> problems, thanks! 
>
> This snippet IMHO deserves a section in the Cookbook: would you please
> like to expand this in a proper section/subsection and submit a patch?

I prefer to keep workarounds for real bugs out of the Cookbook.  For
both fontconfig and the gdm user account’s home directory we should push
a workaround very soon.

>> While we work on fixing these
>
> we in Guix you mean? shouldn't it be fixed upstream?

These problems don’t seem to happen on any distribution.  We still have
to diagnose why that is.

-- 
Ricardo

Re: Stateful system directories

[Giovanni Biscuolo]

Ricardo Wurmus <rekado@elephly.net> writes:

[...]

> I prefer to keep workarounds for real bugs out of the Cookbook.

Even in a specific section called "Workarounds" and with a clear warning
they are just _temporary_ workarounds users could decide to use, while
waiting for upstream to fix that specific issue?

We should not cover every single workaround in the Cookbook, but IMHO
giving context (i.e. explaining the stateful nature of some system
directories giving some example to better understand) and the techniques
used are useful information for Guix System users.

...but I still don't have any useful patch to submit, so I'm just still
speculating

> For both fontconfig and the gdm user account’s home directory we
> should push a workaround very soon.

Fine, I also think we should (this should also be well documented in
comments for each service)

Anyway this thread is not specific to any particurar service or package,
or at least I interpreted it as: "how can Guix System users overcome
occasional configuration statefulness in their systems?", this is the
reason why I gave the other example of Nextcloud and how that issue is
managed in Nix (and IMHO it is far from an ideal solution)

[...]

Thanks! Gio'


-- 
Giovanni Biscuolo

Xelera IT Infrastructures

Re: Stateful system directories

[Ludovic Courtès]

Hello Efraim,

Efraim Flashner <efraim@flashner.co.il> skribis:

> Ignoring the directories in users' home directories, /var/lib/gdm has
> been a source of pain on GNOME upgrades, and we still have some problems
> with /var/cache/fontconfig and I believe there is something else with
> permissions if you switch between ntp and openntpd. I actually have the
> following snippet in my OS-config:
>
> ;; This directory shouldn't exist
> (file-system
>   (device "none")
>   (mount-point "/var/cache/fontconfig")
>   (type "tmpfs")
>   (flags '(read-only))
>   (check? #f))

I think that would work, or we could even make it a writable tmpfs?

(Somehow, I do have /var/cache/fontconfig, but never hard any problems
with it.  It hasn’t been written to in months, and it’s only writable by
root anyway.  Does that mean that people run into problem when they run
GUIs as root?)

> While we work on fixing these does it make sense to modify some of these
> services to unconditionally recreate their home directories on
> boot/activation?

Like /var/lib/gdm?  Maybe.  Or maybe ‘gdm-service-type’ could extend
‘file-system-service-type’ with a tmpfs for /var/lib/gdm?

I suppose that might increase startup time a bit since it’d be
rebuilding its cache every time.  Perhaps we’d also lose bits of state,
no?

Thanks,
Ludo’.

Re: Stateful system directories

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2555 bytes --]

On Sat, Oct 19, 2019 at 11:08:57PM +0200, Ludovic Courtès wrote:
> Hello Efraim,
> 
> Efraim Flashner <efraim@flashner.co.il> skribis:
> 
> > Ignoring the directories in users' home directories, /var/lib/gdm has
> > been a source of pain on GNOME upgrades, and we still have some problems
> > with /var/cache/fontconfig and I believe there is something else with
> > permissions if you switch between ntp and openntpd. I actually have the
> > following snippet in my OS-config:
> >
> > ;; This directory shouldn't exist
> > (file-system
> >   (device "none")
> >   (mount-point "/var/cache/fontconfig")
> >   (type "tmpfs")
> >   (flags '(read-only))
> >   (check? #f))
> 
> I think that would work, or we could even make it a writable tmpfs?

I got angry with it and wanted to see if I could generate any error
messages. :) So far nothing. Of course there isn't a compelling reason
to really make it read-only if we recreate it each time, and it should
cut down on bugs for other directories.

> 
> (Somehow, I do have /var/cache/fontconfig, but never hard any problems
> with it.  It hasn’t been written to in months, and it’s only writable by
> root anyway.  Does that mean that people run into problem when they run
> GUIs as root?)

I have it too, not sure from what. I'm guessing some of the packages
which have fontconfig as an input get a dbus-something to create the
directory if it's missing.

> 
> > While we work on fixing these does it make sense to modify some of these
> > services to unconditionally recreate their home directories on
> > boot/activation?
> 
> Like /var/lib/gdm?  Maybe.  Or maybe ‘gdm-service-type’ could extend
> ‘file-system-service-type’ with a tmpfs for /var/lib/gdm?
> 

Sounds like a good idea. Would that also cause the directory to be
removed if gdm is removed? It should create a tmpfs and mount it over an
existing /var/lib/gdm, right?

> I suppose that might increase startup time a bit since it’d be
> rebuilding its cache every time.  Perhaps we’d also lose bits of state,
> no?

The increase in startup time should be negligible, and according to
rekado, who seems to run into GDM issues the most, removing /var/lib/gdm
is one of the first steps when upgrading gnome or debugging gdm issues.

> 
> Thanks,
> Ludo’.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Stateful system directories

[Ludovic Courtès]

Howdy!

Efraim Flashner <efraim@flashner.co.il> skribis:

> On Sat, Oct 19, 2019 at 11:08:57PM +0200, Ludovic Courtès wrote:
>> Hello Efraim,
>> 
>> Efraim Flashner <efraim@flashner.co.il> skribis:
>> 
>> > Ignoring the directories in users' home directories, /var/lib/gdm has
>> > been a source of pain on GNOME upgrades, and we still have some problems
>> > with /var/cache/fontconfig and I believe there is something else with
>> > permissions if you switch between ntp and openntpd. I actually have the
>> > following snippet in my OS-config:
>> >
>> > ;; This directory shouldn't exist
>> > (file-system
>> >   (device "none")
>> >   (mount-point "/var/cache/fontconfig")
>> >   (type "tmpfs")
>> >   (flags '(read-only))
>> >   (check? #f))
>> 
>> I think that would work, or we could even make it a writable tmpfs?
>
> I got angry with it and wanted to see if I could generate any error
> messages. :) So far nothing. Of course there isn't a compelling reason
> to really make it read-only if we recreate it each time, and it should
> cut down on bugs for other directories.

Yup, let’s do that.

>> (Somehow, I do have /var/cache/fontconfig, but never hard any problems
>> with it.  It hasn’t been written to in months, and it’s only writable by
>> root anyway.  Does that mean that people run into problem when they run
>> GUIs as root?)
>
> I have it too, not sure from what. I'm guessing some of the packages
> which have fontconfig as an input get a dbus-something to create the
> directory if it's missing.

Heh, these dbus things doing stuff behind our back.  :-)

>> > While we work on fixing these does it make sense to modify some of these
>> > services to unconditionally recreate their home directories on
>> > boot/activation?
>> 
>> Like /var/lib/gdm?  Maybe.  Or maybe ‘gdm-service-type’ could extend
>> ‘file-system-service-type’ with a tmpfs for /var/lib/gdm?
>> 
>
> Sounds like a good idea. Would that also cause the directory to be
> removed if gdm is removed? It should create a tmpfs and mount it over an
> existing /var/lib/gdm, right?

Yes.  So the directory won’t be removed if gdm is removed, but that’s
fine, it’ll just be an empty directory sitting there.

>> I suppose that might increase startup time a bit since it’d be
>> rebuilding its cache every time.  Perhaps we’d also lose bits of state,
>> no?
>
> The increase in startup time should be negligible, and according to
> rekado, who seems to run into GDM issues the most, removing /var/lib/gdm
> is one of the first steps when upgrading gnome or debugging gdm issues.

Yeah, it’s a tradeoff, but we should try it on the bare metal to get a
feel.  There’s quite a bit of data in there that we’d be recreating at
each boot:

--8<---------------cut here---------------start------------->8---
$ sudo ls -l /var/lib/gdm/.cache
totalo 16
drwxr-xr-x  2 gdm gdm 4096 Sep 19 08:45 fontconfig
drwxr-xr-x  3 gdm gdm 4096 Apr 11  2019 ibus
drwx------  2 gdm gdm 4096 Apr  1  2019 libgweather
drwxr-xr-x 97 gdm gdm 4096 Sep 19 08:45 mesa_shader_cache
--8<---------------cut here---------------end--------------->8---

If you give it a spin, let us know how it goes!

Ludo’.

Re: Stateful system directories

[Jack Hill]

Today I had an occasion to create a file in gdm's home directory that 
should persist across reboots. I needed to set a dconf setting to 
prevent gdm from putting the computer to sleep. Full details on guix-help 
[0]. Unfortunately, I don't believe there is yet a way to handle these 
setting in a declarative way.

[0] https://lists.gnu.org/archive/html/help-guix/2019-10/msg00213.html

Best,
Jack