bug#45571: Support stable uids and gids for system accounts in a container

bug#45571: Support stable uids and gids for system accounts in a container

[Jason Conroy]

[-- Attachment #1: Type: text/plain, Size: 1098 bytes --]

If I understand correctly, the container script produced by "guix system
container" will allocate the same uid and gid for a service on each
execution, but only if the corresponding entry in the service list has the
same absolute position as it did before. I.e., if the services are
reordered or if there are additions and removals, it's unlikely that the id
allocations will be the same.

As long as a container's filesystems don't outlive the container itself,
this works fine. But when host filesystems are bind-mounted inside the
container with the --share or --expose options, it's important that each
incarnation of a service uses the same uid and gid, because the bind mounts
might be used to hold persistent state for those services.

At first, I thought that I could just define static uids and gids for these
system accounts by adding corresponding user-account and user-group
entries. But this doesn't work: rather than changing how the system
accounts are defined for these services, it results in /etc files with
duplicate entries. (See https://issues.guix.gnu.org/45570 for details.)

[-- Attachment #2: Type: text/html, Size: 1291 bytes --]

bug#45571: Support stable uids and gids for all accounts

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1790 bytes --]

Hi,

I agree that user ids and group ids should be made stable, even in general.

I, too, have been bitten by this.  (So would everyone else if Guix touched
existing UNIX accounts in general)

The right way to make them stable is for Guix ot default each uid to the hash
of the user name.

That said, we'd want to leave free some range of the integer uids for the usual
suspects (yp, samba) to allocate domain users there.

The place to change is gnu/system/accounts.scm.  It would need to be changed
to do something similar for the "uid" field that it already does for the
"home-directory" field.

According to the source code of "useradd" in the package "shadow", it uses
the following range to use for automatic uid assignment:

Range starts at SYS_UID_MIN (default 1) for system user account uids, and stops
at SYS_UID_MAX (default (UID_MIN - 1)).
 
For non-system user account uids, it starts at UID_MIN (default 1000) and
stops at 60000 (UID_MAX).

See /etc/login.defs for the configured values.

Note that Linux has no problem using 32 bit uids.

If we want to make it possible for Guix to distinguish system from non-system
accounts by having different uid ranges for each, "system?" in the
<user-account> record would need to be moved to the front.
Then, in order to be backward compatible, custom procedures/macros
"make-user-account" and "user-account" would need to be provided with the
parameters in the previous order.

Should not be difficult to do--as always, the main work is in agreeing what
should be done, and in testing it after it's done.  The actual change is like
10 lines of source code.

(An easier workaround would be to make the uid mandatory, with the default
being failure.  But that would be the "punting" solution)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

bug#45571: Support stable uids and gids for all accounts

[Jason Conroy]

[-- Attachment #1: Type: text/plain, Size: 2514 bytes --]

Hi Danny,

Your idea has a definite elegance to it. :) I did not realize that Linux
supported 32-bit UIDs out-of-the-box. Still, I wonder if this could
introduce support challenges for packages that incorrectly assume UIDs are
16 bits wide, since they traditionally were that way in UNIX, and since
other Linux distros still seem to prefer small UIDs in their packaging. By
comparison, my earlier idea of declaring static UIDs/GIDs in the
operating-system is decidedly less elegant, but it avoids this particular
risk. Can we be confident that this class of integer width bugs is
extremely rare?

On Fri, Jan 1, 2021 at 9:49 AM Danny Milosavljevic <dannym@scratchpost.org>
wrote:

> Hi,
>
> I agree that user ids and group ids should be made stable, even in general.
>
> I, too, have been bitten by this.  (So would everyone else if Guix touched
> existing UNIX accounts in general)
>
> The right way to make them stable is for Guix ot default each uid to the
> hash
> of the user name.
>
> That said, we'd want to leave free some range of the integer uids for the
> usual
> suspects (yp, samba) to allocate domain users there.
>
> The place to change is gnu/system/accounts.scm.  It would need to be
> changed
> to do something similar for the "uid" field that it already does for the
> "home-directory" field.
>
> According to the source code of "useradd" in the package "shadow", it uses
> the following range to use for automatic uid assignment:
>
> Range starts at SYS_UID_MIN (default 1) for system user account uids, and
> stops
> at SYS_UID_MAX (default (UID_MIN - 1)).
>
> For non-system user account uids, it starts at UID_MIN (default 1000) and
> stops at 60000 (UID_MAX).
>
> See /etc/login.defs for the configured values.
>
> Note that Linux has no problem using 32 bit uids.
>
> If we want to make it possible for Guix to distinguish system from
> non-system
> accounts by having different uid ranges for each, "system?" in the
> <user-account> record would need to be moved to the front.
> Then, in order to be backward compatible, custom procedures/macros
> "make-user-account" and "user-account" would need to be provided with the
> parameters in the previous order.
>
> Should not be difficult to do--as always, the main work is in agreeing what
> should be done, and in testing it after it's done.  The actual change is
> like
> 10 lines of source code.
>
> (An easier workaround would be to make the uid mandatory, with the default
> being failure.  But that would be the "punting" solution)
>

[-- Attachment #2: Type: text/html, Size: 3066 bytes --]

bug#45571: Support stable uids and gids for all accounts

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 2422 bytes --]

Hi Jason,

>Still, I wonder if this could introduce support challenges for packages that
>incorrectly assume UIDs are 16 bits wide, since they traditionally were that
>way in UNIX,

I don't think that these 16 bit problems are common at all since all the
getuid() syscalls I've ever seen, even in traditional UNIXes, have "int" as
return value--and nowadays that's at least 32 bits; and also because UNIX
configuration is usually in text files, which would read and store the stuff
using "%d".  That said, it's possible that it could happen in some seriously
strange configurations (I doubt it).

> and since other Linux distros still seem to prefer small UIDs in
> their packaging. By comparison, my earlier idea of declaring static UIDs/GIDs
> in the operating-system is decidedly less elegant, but it avoids this particular
> risk. 

If we did something like that, I would prefer for guix services to just specify
a fixed UID/GID for each of the entries in gnu/services/*.scm.  We can even
reuse existing system uids and gids that have been statically allocated
already by FreeBSD and Debian and be even uid/gid compatible to those.

For example, change Guix to do (similarly for all Guix services using user
accounts):

diff --git a/gnu/services/monitoring.scm b/gnu/services/monitoring.scm
index 5123a8c441..45d3f4ad17 100644
--- a/gnu/services/monitoring.scm
+++ b/gnu/services/monitoring.scm
@@ -71,6 +71,7 @@
 
 (define %darkstat-accounts
   (list (user-account
+         (uid 4711)
          (name "darkstat")
          (group "darkstat")
          (system? #t)
@@ -78,6 +79,7 @@
          (home-directory "/var/lib/darkstat")
          (shell (file-append shadow "/sbin/nologin")))
         (user-group
+         (id 4711)
          (name "darkstat")
          (system? #t))))

And other constants for other user names.

Or use hashes.  Whichever.  The point is that the ids are deterministic--not
necessarily that we need to use hashes.  The advantage of using a hash is that
we don't need a central registry and this problem will be out of our life once
and for all.

Anyway, as a workaround for your immediate problem, I suggest to preserve
/etc/passwd, /etc/group and /etc/shadow--present entries in those will be
preferred by Guix to even its own configuration.  (I'm not sure whether that's
currently easy to do with guix system container, however)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

bug#45571: Support stable uids and gids for all accounts

[Leo Prikler]

Hello Danny,

I don't think changing the way UIDs are allocated by default is a good
solution as that will break many running installations on real
hardware, that default to those.  A better solution would be to make
user accounts and groups explicit configuration wherever account-
service-type is used, so that it's possible to override them if needed.

Regards,
Leo

bug#45571: Support stable uids and gids for all accounts

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 3198 bytes --]

Hello,

On Fri, 01 Jan 2021 17:20:58 +0100
Leo Prikler <leo.prikler@student.tugraz.at> wrote:

> I don't think changing the way UIDs are allocated by default is a good
> solution as that will break many running installations on real
> hardware, that default to those.  

(gnu build accounts), allocate-passwd defaults to keep using the uids of
existing /etc/passwd entries.  So running installations on real hardware
will not be affected when we change the defaults--otherwise those
installations on real hardware would already have had big problems regarding
user accounts with the current version of Guix when someone adds a user account
(system account or not) to them.  Then, depending on the order of the declared
user-accounts, the uids would have swapped.

>A better solution would be to make
> user accounts and groups explicit configuration wherever account-
> service-type is used, so that it's possible to override them if needed.

They already are explicit.

For example, (gnu services monitoring) has:

>(define %darkstat-accounts
>  (list (user-account
>         (name "darkstat")
>         (group "darkstat")
>         (system? #t)
>         (comment "darkstat daemon user")
>         (home-directory "/var/lib/darkstat")
>         (shell (file-append shadow "/sbin/nologin")))
>        (user-group
>         (name "darkstat")
>         (system? #t))))

and then

>   (extensions
>    (list (service-extension account-service-type
>                             (const %darkstat-accounts))

As you can see, the user and group accounts are already explicit.  What's
more, the uid (and group id) can already be specified right there, and I
argue that it should be specified right there, and that there should be a
stable default if it's not specified.

So to summarize:

(1) This bug is a real problem, and something should be done about it.

(2) The reason it doesn't currently affect Guix system users is because there
is logic preferring existing /etc/passwd, /etc/shadow and /etc/group entries
(which I agree is a good idea).  Doesn't help for guix system container, though.

Or for NFS network file systems.  Any time you have more than one computer
(even with Guix on it) with a filesystem in the network and regular users,
you have to have consistent uids or have a lot of weird uid maps per computer
that someone has to maintain, or worse, an extra service just mapping them.
Why do that to yourself?

(3) For /etc/shadow, it also makes sense to keep the existing entry (the user
name is the key for the search for it btw) because of the password set.

(4) It makes sense to keep the existing /etc/{passwd,shadow,group} entries both
for backward compatibility and also for extensibility of guix with user & group
accounts guix knows nothing about.

(5) Also for not having this bug with containers, it would still be better to
just make uid and gid mandatory for "user-account" records.

(6) Since (5) would move the burden to the user, it would be better usability
to generate uid and gid in a deterministic manner as a default.

Both (5) and (6) can be done even now without problems because of (2).

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

bug#45571: Support stable uids and gids for all accounts

[Leo Prikler]

Hi,

Am Freitag, den 01.01.2021, 18:50 +0100 schrieb Danny Milosavljevic:
> Hello,
> 
> On Fri, 01 Jan 2021 17:20:58 +0100
> Leo Prikler <leo.prikler@student.tugraz.at> wrote:
> 
> > I don't think changing the way UIDs are allocated by default is a
> > good
> > solution as that will break many running installations on real
> > hardware, that default to those.  
> 
> (gnu build accounts), allocate-passwd defaults to keep using the uids
> of
> existing /etc/passwd entries.  So running installations on real
> hardware
> will not be affected when we change the defaults--otherwise those
> installations on real hardware would already have had big problems
> regarding
> user accounts with the current version of Guix when someone adds a
> user account
> (system account or not) to them.  Then, depending on the order of the
> declared
> user-accounts, the uids would have swapped.
Ah, that puts things into perspective.  In other words, the problem is
not, that Guix doesn't read /etc/passwd at all, but that it reads the
wrong one (the host instead of the guest, so to speak).  Should this
perhaps be a parameter instead?

> > A better solution would be to make
> > user accounts and groups explicit configuration wherever account-
> > service-type is used, so that it's possible to override them if
> > needed.
> 
> They already are explicit.
> 
> For example, (gnu services monitoring) has:
> 
> > (define %darkstat-accounts
> >  (list (user-account
> >         (name "darkstat")
> >         (group "darkstat")
> >         (system? #t)
> >         (comment "darkstat daemon user")
> >         (home-directory "/var/lib/darkstat")
> >         (shell (file-append shadow "/sbin/nologin")))
> >        (user-group
> >         (name "darkstat")
> >         (system? #t))))
> 
> and then
> 
> >   (extensions
> >    (list (service-extension account-service-type
> >                             (const %darkstat-accounts))
> 
> As you can see, the user and group accounts are already
> explicit.  What's
> more, the uid (and group id) can already be specified right there,
> and I
> argue that it should be specified right there, and that there should
> be a
> stable default if it's not specified.
How is that explicit?  The % even implies, that it's considered
internal to the definition.  Instead, you'd have (darkstat-accounts
config), which default to the current value of %darkstat-accounts, but
are configurable at least in the way that they allow you to set their
ids.

> So to summarize:
> 
> (1) This bug is a real problem, and something should be done about
> it.
Naturally.

> (2) The reason it doesn't currently affect Guix system users is
> because there
> is logic preferring existing /etc/passwd, /etc/shadow and /etc/group
> entries
> (which I agree is a good idea).  Doesn't help for guix system
> container, though.
> 
> Or for NFS network file systems.  Any time you have more than one
> computer
> (even with Guix on it) with a filesystem in the network and regular
> users,
> you have to have consistent uids or have a lot of weird uid maps per
> computer
> that someone has to maintain, or worse, an extra service just mapping
> them.
> Why do that to yourself?
In the realm of Guix system, this could be resolved by allowing the
user to choose the "seeds" for those files, so to speak (in commands
such as init, vm, deploy, etc.), could it not?
Especially for (3), carrying over the old shadow from the guest rather
than generating a new one with initial passwords sounds like it'd be a
necessary precondition for using them with persistent storage.

> (3) For /etc/shadow, it also makes sense to keep the existing entry
> (the user
> name is the key for the search for it btw) because of the password
> set.
See above.

> (4) It makes sense to keep the existing /etc/{passwd,shadow,group}
> entries both
> for backward compatibility and also for extensibility of guix with
> user & group
> accounts guix knows nothing about.
Not knowing about such accounts sounds somewhat antithetical to Guix,
but whatever.

> (5) Also for not having this bug with containers, it would still be
> better to
> just make uid and gid mandatory for "user-account" records.
> 
> (6) Since (5) would move the burden to the user, it would be better
> usability
> to generate uid and gid in a deterministic manner as a default.
Is the current logic non-deterministic in any way other than supporting
the reuse of old entries (which you yourself agree is a good thing)?
As far as I understand it, same config.scm + same
/etc/{passwd,group,shadow} => same /etc/{passwd,group,shadow}.

Regards,
Leo

bug#45571: Support stable uids and gids for all accounts

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 4609 bytes --]

Hi Leo,

On Fri, 01 Jan 2021 19:44:12 +0100
Leo Prikler <leo.prikler@student.tugraz.at> wrote:

> Ah, that puts things into perspective.  In other words, the problem is
> not, that Guix doesn't read /etc/passwd at all, but that it reads the
> wrong one (the host instead of the guest, so to speak).  Should this
> perhaps be a parameter instead?

Considering the goal of Guix, it's weird that with Guix, one needs to
store&restore /etc/passwd at all.  It's state, but not very useful one.
I mean that's how it is right now--but it's still weird.

With /etc/shadow maybe there's a slightly better case, but note that the key
to find stuff in /etc/shadow can't be the uid--the uid isn't even in there!

> How is that explicit?  The % even implies, that it's considered
> internal to the definition.

Explicit means that the user-account record is initialized right there (every
time account-service-type is extended), by a literal.

And it is.  You can see it plain as day in the guix git repo where
account-service-type is used in gnu/services/ .

Implicit would be if some code would generate this <user-account> record
on-the-fly, usually leaving stuff hard to change by the maintainer.

'(user-account (name "x") ...)' is about as explicit as it gets for a record.

The "%" in the name of the binding changes nothing in the literal value.

And it indeed is possible to add (uid 4711) in the literal and it will work
just fine.

>  Instead, you'd have (darkstat-accounts
> config), which default to the current value of %darkstat-accounts, but
> are configurable at least in the way that they allow you to set their
> ids.

Oh, you want internal service users to be USER-configurable.  Indeed that is
also what Jason suggested in the initial mail.

But I think that that would put undue burden on each user and is really just
a workaround.
I would really like to caution against us doing a "whack a mole" development
approach, where workarounds like that are introduced to work around bugs
without understanding the underlying causes.  So I disagree that having
internal service users be user-configurable is a good idea.

> In the realm of Guix system, this could be resolved by allowing the
> user to choose the "seeds" for those files, so to speak (in commands
> such as init, vm, deploy, etc.), could it not?

Sure, but that's a last resort.  Better is to eliminate state if possible.

> Especially for (3), carrying over the old shadow from the guest rather
> than generating a new one with initial passwords sounds like it'd be a
> necessary precondition for using them with persistent storage.

It depends on what it is used for, really.

> > (5) Also for not having this bug with containers, it would still be
> > better to
> > just make uid and gid mandatory for "user-account" records.
> > 
> > (6) Since (5) would move the burden to the user, it would be better
> > usability
> > to generate uid and gid in a deterministic manner as a default.  

> Is the current logic non-deterministic in any way other than supporting
> the reuse of old entries (which you yourself agree is a good thing)?

It generates uids using a counter, so it depends on what order
user-accounts are created in by Guix, which depends on the order the user
specifies services in /etc/config.scm and on the order to user accounts
are specified in gnu/services/ by guix maintainers.  Then the service
executable (potentially) goes on to create files using those uids.

That means that if you remove or reorder service references in /etc/config.scm,
the uids "want" to change.  The only reason they don't change is because the
logic prefers the existing /etc/passwd's uids--a stopgap measure at the last
second to prevent total chaos.

Does any of this sound good to you?

I mean, strictly speaking, it's better than the alternative--but that's a low
bar.

Better would be a making the uid field mandatory and/or generating each uid
from the respective name.

> As far as I understand it, same config.scm + same
> /etc/{passwd,group,shadow} => same /etc/{passwd,group,shadow}.

That is the intention of (gnu system shadow), I think.  I can't say whether
that's the case in practice now or not.  It certainly was not the case a few
years ago where I did run into the same problem (a necessary condition for
the problem to manifest is that the services change--but my /etc/config.scm
services forms have been stable for a long time now, and Guix upstream also
doesn't change service definitions a lot anymore.  So who knows?).

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

bug#45571: Fwd: Re: bug#45571: Support stable uids and gids for all accounts

[Leo Prikler]

[-- Attachment #1: Type: text/plain, Size: 22 bytes --]

Forgot to CC the ML.

[-- Attachment #2: Weitergeleitete Nachricht – Re: bug#45571: Support stable uids and gids for all accounts --]
[-- Type: message/rfc822, Size: 8125 bytes --]

From: Leo Prikler <leo.prikler@student.tugraz.at>
To: Danny Milosavljevic <dannym@scratchpost.org>
Subject: Re: bug#45571: Support stable uids and gids for all accounts
Date: Sat, 02 Jan 2021 00:16:45 +0100
Message-ID: <58174c197a7b42b29927c492d25e28c684d199ea.camel@student.tugraz.at>

Hi Danny,

Am Freitag, den 01.01.2021, 21:22 +0100 schrieb Danny Milosavljevic:
> Hi Leo,
> 
> On Fri, 01 Jan 2021 19:44:12 +0100
> Leo Prikler <leo.prikler@student.tugraz.at> wrote:
> 
> > Ah, that puts things into perspective.  In other words, the problem
> > is
> > not, that Guix doesn't read /etc/passwd at all, but that it reads
> > the
> > wrong one (the host instead of the guest, so to speak).  Should
> > this
> > perhaps be a parameter instead?
> 
> Considering the goal of Guix, it's weird that with Guix, one needs to
> store&restore /etc/passwd at all.  It's state, but not very useful
> one.
> I mean that's how it is right now--but it's still weird.
> 
> With /etc/shadow maybe there's a slightly better case, but note that
> the key
> to find stuff in /etc/shadow can't be the uid--the uid isn't even in
> there!
AFAIU yes, it's state, but not one that Guix can simply do away with. 
There is not yet a syntax for keeping secrets, which would be needed to
fully populate /etc from config.scm.  Perhaps we'll get there some day.

Also IIUC, account names are supposedly unique, hence my proposed patch
to #45570.

> > How is that explicit?  The % even implies, that it's considered
> > internal to the definition.
> 
> Explicit means that the user-account record is initialized right
> there (every
> time account-service-type is extended), by a literal.
> 
> And it is.  You can see it plain as day in the guix git repo where
> account-service-type is used in gnu/services/ .
> 
> Implicit would be if some code would generate this <user-account>
> record
> on-the-fly, usually leaving stuff hard to change by the maintainer.
Fair enough, that's explicit in some sense, but not explicit if we take
the config.scm as the point of reference.  Since there's no way of
explicitly setting it from there without advanced wizardry.

> '(user-account (name "x") ...)' is about as explicit as it gets for a
> record.
> 
> The "%" in the name of the binding changes nothing in the literal
> value.
> 
> And it indeed is possible to add (uid 4711) in the literal and it
> will work
> just fine.
I'm aware you're joking, or at least I hope you are, but I shouldn't
have to point out why hardcoding ids into those literals is a bad idea.

> >  Instead, you'd have (darkstat-accounts
> > config), which default to the current value of %darkstat-accounts,
> > but
> > are configurable at least in the way that they allow you to set
> > their
> > ids.
> 
> Oh, you want internal service users to be USER-configurable.  Indeed
> that is
> also what Jason suggested in the initial mail.
> 
> But I think that that would put undue burden on each user and is
> really just
> a workaround.
> I would really like to caution against us doing a "whack a mole"
> development
> approach, where workarounds like that are introduced to work around
> bugs
> without understanding the underlying causes.  So I disagree that
> having
> internal service users be user-configurable is a good idea.
"Undue burden on each user" is a bit exaggerated, considering that the
defaults work fine for most.  Making them user-configurable would even
allow hardcoding a default ID, because users would be able to change
that ID if they wanted a different one, eliminating even more reasons
to configure them if not for the reason of "because I want to".

I also disagree, that this is a problem, that we can solve as Guix
System alone.  Even if we were to implement your hash idea, other
distros would still be keeping their numbering systems.  If you share
an NFS with a legacy system, you'd still need a mapping or inherit
accounts from their settings -- neither of which sound great, but
compromises have to be made.

In that sense, I agree with Jason that it is a problem, that you can't
set service [GU]IDs when you want (or need) to.  The question is how to
best make them configurable.

> > In the realm of Guix system, this could be resolved by allowing the
> > user to choose the "seeds" for those files, so to speak (in
> > commands
> > such as init, vm, deploy, etc.), could it not?
> 
> Sure, but that's a last resort.  Better is to eliminate state if
> possible.
Keyword being "if possible".  It is currently not possible to eliminate
shadow.

> > > (5) Also for not having this bug with containers, it would still
> > > be
> > > better to
> > > just make uid and gid mandatory for "user-account" records.
> > > 
> > > (6) Since (5) would move the burden to the user, it would be
> > > better
> > > usability
> > > to generate uid and gid in a deterministic manner as a default.  
> > Is the current logic non-deterministic in any way other than
> > supporting
> > the reuse of old entries (which you yourself agree is a good
> > thing)?
> 
> It generates uids using a counter, so it depends on what order
> user-accounts are created in by Guix, which depends on the order the
> user
> specifies services in /etc/config.scm and on the order to user
> accounts
> are specified in gnu/services/ by guix maintainers.  Then the service
> executable (potentially) goes on to create files using those uids.
> 
> That means that if you remove or reorder service references in
> /etc/config.scm,
> the uids "want" to change.  The only reason they don't change is
> because the
> logic prefers the existing /etc/passwd's uids--a stopgap measure at
> the last
> second to prevent total chaos.
> 
> Does any of this sound good to you?
Keeping value judgements aside, it does sound deterministic.
Relying on order is also not that big of a deal if you have ways of
ensuring order, but you'd also have to ensure, that the expected order
is the same as the one actually used, and that's significantly harder
to do.

An example: You could expect user accounts to be numbered in the order
they appear in the field, but you could also expect them to be numbered
in reverse order (because services can add accounts and they're usually
consed).  In that sense, yeah, relying on order is certainly not
optimal.

> I mean, strictly speaking, it's better than the alternative--but
> that's a low
> bar.
> 
> Better would be a making the uid field mandatory and/or generating
> each uid
> from the respective name.
Statistically speaking hashes collide sooner than incrementing numbers
do.  Granted, they might work as mitigation, but they're no solution on
their own either.

> > As far as I understand it, same config.scm + same
> > /etc/{passwd,group,shadow} => same /etc/{passwd,group,shadow}.
> 
> That is the intention of (gnu system shadow), I think.
You mean (gnu build accounts), right?

> I can't say whether that's the case in practice now or not.  It
> certainly was not the case a few
> years ago where I did run into the same problem (a necessary
> condition for
> the problem to manifest is that the services change--but my
> /etc/config.scm
> services forms have been stable for a long time now, and Guix
> upstream also
> doesn't change service definitions a lot anymore.  So who knows?).
Well, the point here was not to state how outputs differ when the
inputs differ, but what happens, when they stay the same.  I have
another machine, that I carried over from a Gentoo install using
different user names, and there I have a different UID from this
machine, where I did a clean install (and it has stayed the same UID
since).  IOW this appears to be functioning as intended.

Regards,
Leo

bug#45571: Fwd: Re: bug#45571: Support stable uids and gids for all accounts

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1829 bytes --]

Hi Leo,

On Sat, 02 Jan 2021 00:16:45 +0100
Leo Prikler <leo.prikler@student.tugraz.at> wrote:

> > And it indeed is possible to add (uid 4711) in the literal and it
> > will work
> > just fine.  
> I'm aware you're joking, or at least I hope you are, 

What?  It's perfectly reasonable for a distribution to have stable system
user ids.

That's what Debian supports, too:

https://www.debian.org/doc/debian-policy/ch-opersys.html#uid-and-gid-classes

>0-99:
>Globally allocated by the Debian project, the same on every Debian system. These ids will appear in the passwd and group files of all Debian systems, new ids in this range being added automatically as the base-passwd package is updated.

>Packages which need a single statically allocated uid or gid should use one of these; their maintainers should ask the base-passwd maintainer for ids.

[...]

>60000-64999:
>Globally allocated by the Debian project, but only created on demand. The ids are allocated centrally and statically, but the actual accounts are only >created on users’ systems on demand.
>[...]

And so does FreeBSD,
see https://www.freebsd.org/doc/en/books/porters-handbook/users-and-groups.html
and https://github.com/freebsd/freebsd-ports/blob/master/UIDs for the actual registry.

For that matter, IANA does this for ports and many other things.  And so on.

Stable defaults are *good*.

Right now, the Guix service user user-account record specifies 99% of the
/etc/passwd entry.  I indeed propose to make it 100% for system users for Guix
system services.

>but I shouldn't have to point out why hardcoding ids into those literals is a
>bad idea.

You have to point that out to us--especially since Guix service user accounts
of the account-service-type extension can only be instantiated once anyway.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

bug#45571: Fwd: Re: bug#45571: Support stable uids and gids for all accounts

[Leo Prikler]

Hi Danny,
Am Samstag, den 02.01.2021, 02:40 +0100 schrieb Danny Milosavljevic:
> Hi Leo,
> 
> On Sat, 02 Jan 2021 00:16:45 +0100
> Leo Prikler <leo.prikler@student.tugraz.at> wrote:
> 
> > > And it indeed is possible to add (uid 4711) in the literal and it
> > > will work
> > > just fine.  
> > I'm aware you're joking, or at least I hope you are, 
> 
> What?  It's perfectly reasonable for a distribution to have stable
> system
> user ids.
> 
> That's what Debian supports, too:
> 
> https://www.debian.org/doc/debian-policy/ch-opersys.html#uid-and-gid-classes
> 
> > 0-99:
> > Globally allocated by the Debian project, the same on every Debian
> > system. These ids will appear in the passwd and group files of all
> > Debian systems, new ids in this range being added automatically as
> > the base-passwd package is updated.
> > Packages which need a single statically allocated uid or gid should
> > use one of these; their maintainers should ask the base-passwd
> > maintainer for ids.
> 
> [...]
> 
> > 60000-64999:
> > Globally allocated by the Debian project, but only created on
> > demand. The ids are allocated centrally and statically, but the
> > actual accounts are only >created on users’ systems on demand.
> > [...]
You do know, that services such as gdm, pulseaudio, avahi, sshd, mpd,
and others fall into neither region, do you?

> And so does FreeBSD,
> see 
> https://www.freebsd.org/doc/en/books/porters-handbook/users-and-groups.html
> and https://github.com/freebsd/freebsd-ports/blob/master/UIDs for the
> actual registry.
If I had a guixbuilder for every account in that list, that I didn't
need, I'd have a lot of guixbuilders.  Probably more than I could
allocate into a contiguous block under FreeBSD.

> For that matter, IANA does this for ports and many other things.  And
> so on.
> 
> Stable defaults are *good*.
So is leaving room for other configurations.  Some of the bindings we
now consider "default" were only made because other ports were already
claimed.  Not to mention overlaps, such as port 465.

> Right now, the Guix service user user-account record specifies 99% of
> the
> /etc/passwd entry.  I indeed propose to make it 100% for system users
> for Guix
> system services.
What's the remaining 1%?

> > but I shouldn't have to point out why hardcoding ids into those
> > literals is a
> > bad idea.
> 
> You have to point that out to us--especially since Guix service user
> accounts
> of the account-service-type extension can only be instantiated once
> anyway.
Unlike in other systems, where you'd expect people to manually fiddle
around with such files and tragically fail, in Guix your OS config.scm
should reflect the actual state of the system (modulo secrets, that
can't be expressed currently).  If you claim UID 92 for GDM like
FreeBSD does, but people live on installations, that have the old
default of 983 (or any other, depending on the number of guixbuilders
you have), that's going to cause problems.  Perhaps not the same
problems that led to the creation of its activation-service, but still.

That's not to say, that claiming such IDs is *always* bad, just that
it's bad to do so without leaving room for configuration.  I should
likely have worded that better, but at the same time there was context
from which one could have inferred, that I meant hardcoding IDs into
unchanging constants.

From the solutions we do have so far, I believe that making user
accounts an explicit part of service configuration (in what shape may
still be up for debate), with reasonable defaults including numeric
UIDs and GIDs (at least) for essential services such as GDM sounds like
the best option.  WDYT?

Regards,
Leo

bug#45571: Fwd: Re: bug#45571: Support stable uids and gids for all accounts

[Jason Conroy]

[-- Attachment #1: Type: text/plain, Size: 1525 bytes --]

On Fri, Jan 1, 2021 at 10:11 PM Leo Prikler <leo.prikler@student.tugraz.at>
wrote:

> Hi Danny,
> Am Samstag, den 02.01.2021, 02:40 +0100 schrieb Danny Milosavljevic:
> > Hi Leo,
> >
> > On Sat, 02 Jan 2021 00:16:45 +0100
> > Leo Prikler <leo.prikler@student.tugraz.at> wrote:
> >
> > > > And it indeed is possible to add (uid 4711) in the literal and it
> > > > will work
> > > > just fine.
> > > I'm aware you're joking, or at least I hope you are,
> >
> > What?  It's perfectly reasonable for a distribution to have stable
> > system
> > user ids.
>

My reaction to this was not that defaults are bad, but that dispersing
numeric literals throughout the code is. Collectively these values specify
the contents of a registry, so that registry might as well be located
centrally. Or at least, there should be some mechanism to ensure that two
services can't claim the same default ID, otherwise the collision will not
manifest until somebody instantiates a system with the colliding services.

From the solutions we do have so far, I believe that making user
> accounts an explicit part of service configuration (in what shape may
> still be up for debate), with reasonable defaults including numeric
> UIDs and GIDs (at least) for essential services such as GDM sounds like
> the best option.  WDYT?
>
> Regards,
> Leo
>

That seems reasonable to me. As for representation, I think there's value
decoupling these settings from a service's own config so that support for
custom UIDs/GIDs remains consistent across services.

[-- Attachment #2: Type: text/html, Size: 2317 bytes --]

bug#45571: Fwd: Re: bug#45571: Support stable uids and gids for all accounts

[Leo Prikler]

Hi Jason,
Am Samstag, den 02.01.2021, 09:02 -0500 schrieb Jason Conroy:
> On Fri, Jan 1, 2021 at 10:11 PM Leo Prikler <
> leo.prikler@student.tugraz.at> wrote:
> > Hi Danny,
> > Am Samstag, den 02.01.2021, 02:40 +0100 schrieb Danny
> > Milosavljevic:
> > > Hi Leo,
> > > 
> > > On Sat, 02 Jan 2021 00:16:45 +0100
> > > Leo Prikler <leo.prikler@student.tugraz.at> wrote:
> > > 
> > > > > And it indeed is possible to add (uid 4711) in the literal
> > and it
> > > > > will work
> > > > > just fine.  
> > > > I'm aware you're joking, or at least I hope you are, 
> > > 
> > > What?  It's perfectly reasonable for a distribution to have
> > stable
> > > system
> > > user ids.
> 
> My reaction to this was not that defaults are bad, but that
> dispersing numeric literals throughout the code is. Collectively
> these values specify the contents of a registry, so that registry
> might as well be located centrally. Or at least, there should be some
> mechanism to ensure that two services can't claim the same default
> ID, otherwise the collision will not manifest until somebody
> instantiates a system with the colliding services.
I think it would suffice to raise a condition from shadow.scm, much
like my proposed fix for #45570.  As far as development is concerned,
one could add a check to see, that no conflicts exist between services
extending account-service-type.

> > From the solutions we do have so far, I believe that making user
> > accounts an explicit part of service configuration (in what shape
> > may
> > still be up for debate), with reasonable defaults including numeric
> > UIDs and GIDs (at least) for essential services such as GDM sounds
> > like
> > the best option.  WDYT?
> > 
> > Regards,
> > Leo
> 
> That seems reasonable to me. As for representation, I think there's
> value decoupling these settings from a service's own config so that
> support for custom UIDs/GIDs remains consistent across services.
In this case I'd agree with Danny, that asking users to update two
fields to get one service working puts an excessive burden on them.  To
be fair, I don't even necessarily think, that making the full account
configurable is a good idea, unless someone wants to argue, that
there's value in potentially giving those accounts shell access.  At
the very least, there should be syntax like

(user-account
 (inherit sane-system-account-defaults)
 (name "gdm")
 (id 92))

Perhaps there could also be a procedure (system-account+group name
#:key comment uid gid).

Regards,
Leo

bug#45571: Fwd: Re: bug#45571: Support stable uids and gids for all accounts

[Jason Conroy]

[-- Attachment #1: Type: text/plain, Size: 3701 bytes --]

On Sat, Jan 2, 2021 at 9:29 AM Leo Prikler <leo.prikler@student.tugraz.at>
wrote:

> Hi Jason,
> Am Samstag, den 02.01.2021, 09:02 -0500 schrieb Jason Conroy:
> > On Fri, Jan 1, 2021 at 10:11 PM Leo Prikler <
> > leo.prikler@student.tugraz.at> wrote:
> > > Hi Danny,
> > > Am Samstag, den 02.01.2021, 02:40 +0100 schrieb Danny
> > > Milosavljevic:
> > > > Hi Leo,
> > > >
> > > > On Sat, 02 Jan 2021 00:16:45 +0100
> > > > Leo Prikler <leo.prikler@student.tugraz.at> wrote:
> > > >
> > > > > > And it indeed is possible to add (uid 4711) in the literal
> > > and it
> > > > > > will work
> > > > > > just fine.
> > > > > I'm aware you're joking, or at least I hope you are,
> > > >
> > > > What?  It's perfectly reasonable for a distribution to have
> > > stable
> > > > system
> > > > user ids.
> >
> > My reaction to this was not that defaults are bad, but that
> > dispersing numeric literals throughout the code is. Collectively
> > these values specify the contents of a registry, so that registry
> > might as well be located centrally. Or at least, there should be some
> > mechanism to ensure that two services can't claim the same default
> > ID, otherwise the collision will not manifest until somebody
> > instantiates a system with the colliding services.
> I think it would suffice to raise a condition from shadow.scm, much
> like my proposed fix for #45570.  As far as development is concerned,
> one could add a check to see, that no conflicts exist between services
> extending account-service-type.
>

Assuming that authors of new services tend to choose the lowest free ID, is
this validation sufficient to ensure that two services checked around the
same time by different authors won't collide?

>
> > > From the solutions we do have so far, I believe that making user
> > > accounts an explicit part of service configuration (in what shape
> > > may
> > > still be up for debate), with reasonable defaults including numeric
> > > UIDs and GIDs (at least) for essential services such as GDM sounds
> > > like
> > > the best option.  WDYT?
> > >
> > > Regards,
> > > Leo
> >
> > That seems reasonable to me. As for representation, I think there's
> > value decoupling these settings from a service's own config so that
> > support for custom UIDs/GIDs remains consistent across services.
>
In this case I'd agree with Danny, that asking users to update two
> fields to get one service working puts an excessive burden on them.


Leo, I'm not sure what you mean by updating two fields. Are you saying that
some services already permit manual selection of UIDs? I was proposing
setting that value in the "users" section of operating-system (or
elsewhere) rather than setting it in the "services" section, but not both
places.

Since Guix already uses a central allocator for UIDs and GIDs (implemented
using simple counters), I was imagining a model where the decision is still
made centrally, but with different inputs: 1) a global mapping from
user/group name to default ID; and 2) a similar name-to-ID mapping in
operating-system where users specify their overrides.

>
> To
> be fair, I don't even necessarily think, that making the full account
> configurable is a good idea, unless someone wants to argue, that
> there's value in potentially giving those accounts shell access.


The thought of making other parts of the account configurable occurred to
me, but I couldn't come up with any serious use cases either.


> At
> the very least, there should be syntax like
>
> (user-account
>  (inherit sane-system-account-defaults)
>  (name "gdm")
>  (id 92))
>
> Perhaps there could also be a procedure (system-account+group name
> #:key comment uid gid).
>
> Regards,
> Leo
>
>

[-- Attachment #2: Type: text/html, Size: 5457 bytes --]

bug#45571: Fwd: Re: bug#45571: Support stable uids and gids for all accounts

[Leo Prikler]

Hello Jason,
Am Samstag, den 02.01.2021, 09:52 -0500 schrieb Jason Conroy:
> On Sat, Jan 2, 2021 at 9:29 AM Leo Prikler <
> leo.prikler@student.tugraz.at> wrote:
> > Hi Jason,
> > Am Samstag, den 02.01.2021, 09:02 -0500 schrieb Jason Conroy:
> > > On Fri, Jan 1, 2021 at 10:11 PM Leo Prikler <
> > > leo.prikler@student.tugraz.at> wrote:
> > > > Hi Danny,
> > > > Am Samstag, den 02.01.2021, 02:40 +0100 schrieb Danny
> > > > Milosavljevic:
> > > > > Hi Leo,
> > > > > 
> > > > > On Sat, 02 Jan 2021 00:16:45 +0100
> > > > > Leo Prikler <leo.prikler@student.tugraz.at> wrote:
> > > > > 
> > > > > > > And it indeed is possible to add (uid 4711) in the
> > literal
> > > > and it
> > > > > > > will work
> > > > > > > just fine.  
> > > > > > I'm aware you're joking, or at least I hope you are, 
> > > > > 
> > > > > What?  It's perfectly reasonable for a distribution to have
> > > > stable
> > > > > system
> > > > > user ids.
> > > 
> > > My reaction to this was not that defaults are bad, but that
> > > dispersing numeric literals throughout the code is. Collectively
> > > these values specify the contents of a registry, so that registry
> > > might as well be located centrally. Or at least, there should be
> > some
> > > mechanism to ensure that two services can't claim the same
> > default
> > > ID, otherwise the collision will not manifest until somebody
> > > instantiates a system with the colliding services.
> > I think it would suffice to raise a condition from shadow.scm, much
> > like my proposed fix for #45570.  As far as development is
> > concerned,
> > one could add a check to see, that no conflicts exist between
> > services
> > extending account-service-type.
> 
> Assuming that authors of new services tend to choose the lowest free
> ID, is this validation sufficient to ensure that two services checked
> around the same time by different authors won't collide?
No, you'd need to lint the services in the pre-push hook.  That would
not be the biggest deal though, we already authenticate the commits and
check whether the NEWS file is broken before pushing, for instance.  I
believe it could also be checked without actually instantiating that
system, but don't quote me on that.

> > > > From the solutions we do have so far, I believe that making
> > user
> > > > accounts an explicit part of service configuration (in what
> > shape
> > > > may
> > > > still be up for debate), with reasonable defaults including
> > numeric
> > > > UIDs and GIDs (at least) for essential services such as GDM
> > sounds
> > > > like
> > > > the best option.  WDYT?
> > > > 
> > > > Regards,
> > > > Leo
> > > 
> > > That seems reasonable to me. As for representation, I think
> > there's
> > > value decoupling these settings from a service's own config so
> > that
> > > support for custom UIDs/GIDs remains consistent across services. 
> > In this case I'd agree with Danny, that asking users to update two
> > fields to get one service working puts an excessive burden on
> > them.  
> 
> Leo, I'm not sure what you mean by updating two fields. Are you
> saying that some services already permit manual selection of UIDs? I
> was proposing setting that value in the "users" section of operating-
> system (or elsewhere) rather than setting it in the "services"
> section, but not both places.
As far as I'm aware, no service so far do that, but there are some,
that don't set up the user (e.g. mpd).  However, I don't think, that's
the way to go for services like gdm.  If you decoupled the gdm user and
group from its service specification, you'd need to modify three
operating-system fields to add gdm as opposed to one.  If the gdm user
and group were configuration, you could instead specify them with
modify-services or gdm-service-type itself.

> Since Guix already uses a central allocator for UIDs and GIDs
> (implemented using simple counters), I was imagining a model where
> the decision is still made centrally, but with different inputs: 1) a
> global mapping from user/group name to default ID; and 2) a similar
> name-to-ID mapping in operating-system where users specify their
> overrides.
I'm not sure how well that'd work together with account-service-type. 
You'd have to find a novel way of extending it, that's for sure.

> > To
> > be fair, I don't even necessarily think, that making the full
> > account
> > configurable is a good idea, unless someone wants to argue, that
> > there's value in potentially giving those accounts shell access.  
> 
> The thought of making other parts of the account configurable
> occurred to me, but I couldn't come up with any serious use cases
> either.
As far as I'm aware, nologin exists for a good reason.

Regards,
Leo

bug#45571: Fwd: Re: bug#45571: Support stable uids and gids for all accounts

[Jason Conroy]

[-- Attachment #1: Type: text/plain, Size: 5282 bytes --]

Hi Leo,

On Sat, Jan 2, 2021 at 10:35 AM Leo Prikler <leo.prikler@student.tugraz.at>
wrote:

> Hello Jason,
> Am Samstag, den 02.01.2021, 09:52 -0500 schrieb Jason Conroy:
> > On Sat, Jan 2, 2021 at 9:29 AM Leo Prikler <
> > leo.prikler@student.tugraz.at> wrote:
> > > Hi Jason,
> > > Am Samstag, den 02.01.2021, 09:02 -0500 schrieb Jason Conroy:
> > > > On Fri, Jan 1, 2021 at 10:11 PM Leo Prikler <
> > > > leo.prikler@student.tugraz.at> wrote:
> > > > > Hi Danny,
> > > > > Am Samstag, den 02.01.2021, 02:40 +0100 schrieb Danny
> > > > > Milosavljevic:
> > > > > > Hi Leo,
> > > > > >
> > > > > > On Sat, 02 Jan 2021 00:16:45 +0100
> > > > > > Leo Prikler <leo.prikler@student.tugraz.at> wrote:
> > > > > >
> > > > > > > > And it indeed is possible to add (uid 4711) in the
> > > literal
> > > > > and it
> > > > > > > > will work
> > > > > > > > just fine.
> > > > > > > I'm aware you're joking, or at least I hope you are,
> > > > > >
> > > > > > What?  It's perfectly reasonable for a distribution to have
> > > > > stable
> > > > > > system
> > > > > > user ids.
> > > >
> > > > My reaction to this was not that defaults are bad, but that
> > > > dispersing numeric literals throughout the code is. Collectively
> > > > these values specify the contents of a registry, so that registry
> > > > might as well be located centrally. Or at least, there should be
> > > some
> > > > mechanism to ensure that two services can't claim the same
> > > default
> > > > ID, otherwise the collision will not manifest until somebody
> > > > instantiates a system with the colliding services.
> > > I think it would suffice to raise a condition from shadow.scm, much
> > > like my proposed fix for #45570.  As far as development is
> > > concerned,
> > > one could add a check to see, that no conflicts exist between
> > > services
> > > extending account-service-type.
> >
> > Assuming that authors of new services tend to choose the lowest free
> > ID, is this validation sufficient to ensure that two services checked
> > around the same time by different authors won't collide?
> No, you'd need to lint the services in the pre-push hook.  That would
> not be the biggest deal though, we already authenticate the commits and
> check whether the NEWS file is broken before pushing, for instance.  I
> believe it could also be checked without actually instantiating that
> system, but don't quote me on that.
>

Ok, that seems achievable. I would only point out that with a central UID
registry you get that validation "for free" in the form of a Git merge
conflict.


> > > > > From the solutions we do have so far, I believe that making
> > > user
> > > > > accounts an explicit part of service configuration (in what
> > > shape
> > > > > may
> > > > > still be up for debate), with reasonable defaults including
> > > numeric
> > > > > UIDs and GIDs (at least) for essential services such as GDM
> > > sounds
> > > > > like
> > > > > the best option.  WDYT?
> > > > >
> > > > > Regards,
> > > > > Leo
> > > >
> > > > That seems reasonable to me. As for representation, I think
> > > there's
> > > > value decoupling these settings from a service's own config so
> > > that
> > > > support for custom UIDs/GIDs remains consistent across services.
> > > In this case I'd agree with Danny, that asking users to update two
> > > fields to get one service working puts an excessive burden on
> > > them.
> >
> > Leo, I'm not sure what you mean by updating two fields. Are you
> > saying that some services already permit manual selection of UIDs? I
> > was proposing setting that value in the "users" section of operating-
> > system (or elsewhere) rather than setting it in the "services"
> > section, but not both places.
> As far as I'm aware, no service so far do that, but there are some,
> that don't set up the user (e.g. mpd).  However, I don't think, that's
> the way to go for services like gdm.  If you decoupled the gdm user and
> group from its service specification, you'd need to modify three
> operating-system fields to add gdm as opposed to one.  If the gdm user
> and group were configuration, you could instead specify them with
> modify-services or gdm-service-type itself.
>
> > Since Guix already uses a central allocator for UIDs and GIDs
> > (implemented using simple counters), I was imagining a model where
> > the decision is still made centrally, but with different inputs: 1) a
> > global mapping from user/group name to default ID; and 2) a similar
> > name-to-ID mapping in operating-system where users specify their
> > overrides.
> I'm not sure how well that'd work together with account-service-type.
> You'd have to find a novel way of extending it, that's for sure.
>
> > > To
> > > be fair, I don't even necessarily think, that making the full
> > > account
> > > configurable is a good idea, unless someone wants to argue, that
> > > there's value in potentially giving those accounts shell access.
> >
> > The thought of making other parts of the account configurable
> > occurred to me, but I couldn't come up with any serious use cases
> > either.
> As far as I'm aware, nologin exists for a good reason.
>

No arguments there. :) I thought your point was that we don't have a
compelling reason to let the end user replace it with something else.

[-- Attachment #2: Type: text/html, Size: 7233 bytes --]

bug#45571: Fwd: Re: bug#45571: Support stable uids and gids for all accounts

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 4173 bytes --]

Hi Jason,

On Sat, 2 Jan 2021 09:02:18 -0500
Jason Conroy <conjaroy@gmail.com> wrote:

> My reaction to this was not that defaults are bad, but that dispersing
> numeric literals throughout the code is. 

In general that is not exactly true.  What you want is some way to check
the uids for collisions--and putting them into one file only and manually
checking is only one way to do that.  And it's not ideal because then the uid
to use for an account would be in some random file far away from the actual
point of use.  If you mean having both the central registry, and the numeric
literal throughout the code, carry on--I agree.

We also disperse shepherd service names and many other similar things across
literals in the source code of Guix.  Those can cause similar problems.

I agree that it would be better to have a checker, and central registry, for
cross-checking--but right now we don't do that for a lot of other things,
among which are the shepherd service names, dbus service names and dbus
interface names.

This is guile--it could find and lint user account definitions perfectly well,
no matter where they are.  There could be an automated check that the uids are
not duplicated, at compile or lint time.  It would be nice to have such a
checker for the dbus things, too.

But seriously, at this point I don't think any of this in the
currently-discussed form adds enough value to be worth the complexity
and the risk of changing it in the first place.

>Collectively these values specify
> the contents of a registry, so that registry might as well be located
> centrally. 

I agree, if in addition.

(Or if a registry is undesireable, calculate the uids by user name.
These are the possibilities.  If hashing user names is too uncommon on
UNIX elsewhere, I say that we have no /usr or /lib, so we are not exactly doing
common things in Guix because they are common--what we do really depends on the
merits)

>Or at least, there should be some mechanism to ensure that two
> services can't claim the same default ID, otherwise the collision will not
> manifest until somebody instantiates a system with the colliding services.

I agree--that mechanism would need to be added.

> >From the solutions we do have so far, I believe that making user
> > accounts an explicit part of service configuration (in what shape may
> > still be up for debate), 

Not everything needs to be user configurable.  You suggest having these kinds
of things especially user-configurable as a work-around for them not being
stable, right?  Or do you want it in general?

I would like to avoid them here, if only needed for that reason.

(Also, if they are user-configurable, then it's much easier for uid collisions
to happen than if Guix manages them)

So after thinking about it some more, I disagree that that is the best option
for this specific problem.

In my opinion, the best option here is all of these things:

(1) To document that you need to seed /etc/passwd (for Docker etc)
(2) For guix system container to default to the host's /etc/passwd and
(3) For guix system container to add and retain (!) its own /etc/passwd for
accounts only it has (merged together with the host's /etc/passwd for ease
of implementation)

The suggestions I made before that would obsolete /etc/passwd storage got
watered down to a version where they don't obsolete /etc/passwd storage and
thus I oppose doing them in that form.  It would be making the stuff more
complicated--and now you'd have TWO /etc/passwd:

* one /etc/template-passwd (for the uid registry)
(might as well integrate that into guix as a scheme file, though),
* and one /etc/passwd with the currently created users. 

This seems to be excessive just for making uids stable.

> That seems reasonable to me. As for representation, I think there's value
> decoupling these settings from a service's own config so that support for
> custom UIDs/GIDs remains consistent across services.

Is there a need for using custom service UIDs?
I think if so, that is independent of this bug report, which asked for
stable UIDs and GIDs.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

bug#45571: Fwd: Re: bug#45571: Support stable uids and gids for all accounts

[Jason Conroy]

[-- Attachment #1: Type: text/plain, Size: 1005 bytes --]

Hi Danny,

On Sat, Jan 2, 2021 at 9:50 AM Danny Milosavljevic <dannym@scratchpost.org>
wrote:

> > >From the solutions we do have so far, I believe that making user
> > > accounts an explicit part of service configuration (in what shape may
> > > still be up for debate),
>
> Not everything needs to be user configurable.  You suggest having these
> kinds
> of things especially user-configurable as a work-around for them not being
> stable, right?  Or do you want it in general?
>
> I would like to avoid them here, if only needed for that reason.
>
> Is there a need for using custom service UIDs?
> I think if so, that is independent of this bug report, which asked for
> stable UIDs and GIDs.
>

As far as I know, stability is sufficient for me, but Leo raised some cases
where specific values would ease migration. An /etc/passwd seed file seems
like another reasonable way to handle that, but I haven't investigated how
difficult this change would be compared adding new knobs in
operating-system.

[-- Attachment #2: Type: text/html, Size: 1393 bytes --]

bug#45571: Fwd: Re: bug#45571: Support stable uids and gids for all accounts

[Leo Prikler]

Hi Danny,
Am Samstag, den 02.01.2021, 15:50 +0100 schrieb Danny Milosavljevic:
> [...]
> I agree, if in addition.
> 
> (Or if a registry is undesireable, calculate the uids by user name.
> These are the possibilities.  If hashing user names is too uncommon
> on
> UNIX elsewhere, I say that we have no /usr or /lib, so we are not
> exactly doing
> common things in Guix because they are common--what we do really
> depends on the
> merits)
I don't think hashing has much merit if you have the range of 100-1000
to work with.  Using the full 32 bit integer range also feels like a
hack just to enable hashing, and even then hash functions targeting 32
bit integers are not known to be the most secure in this world.  In
other words, hashing user names into IDs is little more than theatre
and while it can certainly mitigate the underlying issue in *some*
cases, it is not by itself a solution.

> > > From the solutions we do have so far, I believe that making user
> > > accounts an explicit part of service configuration (in what shape
> > > may
> > > still be up for debate), 
> 
> Not everything needs to be user configurable.  You suggest having
> these kinds
> of things especially user-configurable as a work-around for them not
> being
> stable, right?  Or do you want it in general?
I believe there might be a general utility for that if we're not going
to use an existing passwd as oracle.  In that case, you would need a
way of claiming those IDs from the config.scm.

Other than that, I believe you pointed out an NFS example, where it
could also be beneficial to sync up user/system accounts with the IDs
they have on other machines within the network.  Of course, if all
machines within the network use Guix, that point is moot, but there
might be a case when you want to play nice with that one old Debian
server.

> I would like to avoid them here, if only needed for that reason.
> 
> (Also, if they are user-configurable, then it's much easier for uid
> collisions
> to happen than if Guix manages them)
Sure, but either way we'll need a collision checker, will we not?

> So after thinking about it some more, I disagree that that is the
> best option
> for this specific problem.
> 
> In my opinion, the best option here is all of these things:
> 
> (1) To document that you need to seed /etc/passwd (for Docker etc)
> (2) For guix system container to default to the host's /etc/passwd
> and
> (3) For guix system container to add and retain (!) its own
> /etc/passwd for
> accounts only it has (merged together with the host's /etc/passwd for
> ease
> of implementation)
That's also a solution and I think I'd prefer that over spamming IDs
throughout the codebase.

> The suggestions I made before that would obsolete /etc/passwd storage
> got
> watered down to a version where they don't obsolete /etc/passwd
> storage and
> thus I oppose doing them in that form.  It would be making the stuff
> more
> complicated--and now you'd have TWO /etc/passwd:
> 
> * one /etc/template-passwd (for the uid registry)
> (might as well integrate that into guix as a scheme file, though),
> * and one /etc/passwd with the currently created users. 
> 
> This seems to be excessive just for making uids stable.
FWIW I think taking the passwd-seed as a file-like object, that
defaults to using the host's /etc/passwd might work here, but I'm not
completely sure.

> > That seems reasonable to me. As for representation, I think there's
> > value
> > decoupling these settings from a service's own config so that
> > support for
> > custom UIDs/GIDs remains consistent across services.
> 
> Is there a need for using custom service UIDs?
> I think if so, that is independent of this bug report, which asked
> for
> stable UIDs and GIDs.
I agree, the discourse regarding that has been muddled a bit, but since
custom implies stable I don't think this option can be completely
discarded.  Of course, both stability and customization are also given
by a passwd-seed, so if we choose that implementation, other means of
customization might not be needed.

Regards,
Leo

bug#45571: Support stable uids and gids for all accounts

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 4672 bytes --]

Hi Leo,

On Fri, 01 Jan 2021 19:44:12 +0100
Leo Prikler <leo.prikler@student.tugraz.at> wrote:

> Ah, that puts things into perspective.  In other words, the problem is
> not, that Guix doesn't read /etc/passwd at all, but that it reads the
> wrong one (the host instead of the guest, so to speak).  Should this
> perhaps be a parameter instead?  

Considering the goal of Guix, it's weird that with Guix, one needs to
store&restore /etc/passwd at all.  It's state, but not very useful one.
I mean that's how it is right now--but it's still weird.

With /etc/shadow maybe there's a slightly better case, but note that the key
to find stuff in /etc/shadow can't be the uid--the uid isn't even in there!

> How is that explicit?  The % even implies, that it's considered
> internal to the definition.  

Explicit means that the user-account record is initialized right there (every
time account-service-type is extended), by a literal.

And it is.  You can see it plain as day in the guix git repo where
account-service-type is used in gnu/services/ .

Implicit would be if some code would generate this <user-account> record
on-the-fly, usually leaving stuff hard to change by the maintainer.

'(user-account (name "x") ...)' is about as explicit as it gets for a record.

The "%" in the name of the binding changes nothing in the literal value.

And it indeed is possible to add (uid 4711) in the literal and it will work
just fine.

>  Instead, you'd have (darkstat-accounts
> config), which default to the current value of %darkstat-accounts, but
> are configurable at least in the way that they allow you to set their
> ids.  

Oh, you want internal service users to be USER-configurable.  Indeed that is
also what Jason suggested in the initial mail.

But I think that that would put undue burden on each user and is really just
a workaround.
I would really like to caution against us doing a "whack a mole" development
approach, where workarounds like that are introduced to work around bugs
without understanding the underlying causes.  So I disagree that having
internal service users be user-configurable is a good idea.

> In the realm of Guix system, this could be resolved by allowing the
> user to choose the "seeds" for those files, so to speak (in commands
> such as init, vm, deploy, etc.), could it not?  

Sure, but that's a last resort.  Better is to eliminate state if possible.

> Especially for (3), carrying over the old shadow from the guest rather
> than generating a new one with initial passwords sounds like it'd be a
> necessary precondition for using them with persistent storage.  

It depends on what it is used for, really.

> > (5) Also for not having this bug with containers, it would still be
> > better to
> > just make uid and gid mandatory for "user-account" records.
> > 
> > (6) Since (5) would move the burden to the user, it would be better
> > usability
> > to generate uid and gid in a deterministic manner as a default.    

> Is the current logic non-deterministic in any way other than supporting
> the reuse of old entries (which you yourself agree is a good thing)?  

It generates uids using a counter, so it depends on what order
user-accounts are created in by Guix, which depends on the order the user
specifies services in /etc/config.scm and on the order to user accounts
are specified in gnu/services/ by guix maintainers.  Then the service
executable (potentially) goes on to create files using those uids.

That means that if you remove or reorder service references in /etc/config.scm,
the uids "want" to change.  The only reason they don't change is because the
logic prefers the existing /etc/passwd's uids--a stopgap measure at the last
second to prevent total chaos.

Does any of this sound good to you?

I mean, strictly speaking, it's better than the alternative--but that's a low
bar.

Better would be a making the uid field mandatory and/or generating each uid
from the respective name.

> As far as I understand it, same config.scm + same
> /etc/{passwd,group,shadow} => same /etc/{passwd,group,shadow}.  

That is the intention of (gnu system shadow), I think.  I can't say whether
that's the case in practice now or not.  It certainly was not the case a few
years ago where I did run into the same problem (a necessary condition for
the problem to manifest is that the services change--but my /etc/config.scm
services forms have been stable for a long time now, and Guix upstream also
doesn't change service definitions a lot anymore.  So who knows?).


-- 
W: https://www.friendly-machines.at/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

bug#45571: Support stable uids and gids for system accounts in a container

[Brendan Tildesley via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 387 bytes --]

It may be useful to see what Nix does.

Nix has many static id, with the comment:

# IMPORTANT!
# We only add static uids and gids for services where it is not feasible
# to change uids/gids on service start, in example a service with a lot of
# files. Please also check if the service is applicable for systemd's

https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/misc/ids.nix

[-- Attachment #2: Type: text/html, Size: 1211 bytes --]

bug#45571: Support stable uids and gids for system accounts in a container

[Arun Isaac]

[-- Attachment #1: Type: text/plain, Size: 246 bytes --]


Hi all,

For many services, extending the activation-service-type with a gexp
that can chown the necessary files is a good enough solution. This is
how I work around the problem of unstable uids/gids in my guix system
containers.

Regards,
Arun

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 524 bytes --]

bug#45571: Support stable uids and gids for all accounts

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1395 bytes --]

Hi Leo,

> > Considering the goal of Guix, it's weird that with Guix, one needs to
> > store&restore /etc/passwd at all.  It's state, but not very useful
> > one.
> > I mean that's how it is right now--but it's still weird.
> > With /etc/shadow maybe there's a slightly better case, but note that
> > the key
> > to find stuff in /etc/shadow can't be the uid--the uid isn't even in
> > there!  

> AFAIU yes, it's state, but not one that Guix can simply do away with. 

It's easily possible to recreate /etc/passwd from scratch if the uids are
always specified in <user-account>s and thus /etc/passwd would not need to
be persistent state anymore.  Right now everything from /etc/passwd except
the uid and the comment is already specified in <user-account>.

So Guix can indeed simply do away with the persistent state of
/etc/passwd--that's why I suggested specifying the uids in the first place.

(By now I don't think that's the best way to make UIDs stable, but it's
factually incorrect to assert that Guix can't simply do away with that
persistent state specifically.  It can.)

> There is not yet a syntax for keeping secrets, which would be needed to
> fully populate /etc from config.scm.  Perhaps we'll get there some day.

/etc/passwd does not contain secrets.  Neither does /etc/group.

And /etc/shadow doesn't contain uids.

So there is no conflict.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

bug#45571: Support stable uids and gids for all accounts

[Leo Prikler]

Hi Danny,
Am Samstag, den 02.01.2021, 16:04 +0100 schrieb Danny Milosavljevic:
> Hi Leo,
> 
> > > Considering the goal of Guix, it's weird that with Guix, one
> > > needs to
> > > store&restore /etc/passwd at all.  It's state, but not very
> > > useful
> > > one.
> > > I mean that's how it is right now--but it's still weird.
> > > With /etc/shadow maybe there's a slightly better case, but note
> > > that
> > > the key
> > > to find stuff in /etc/shadow can't be the uid--the uid isn't even
> > > in
> > > there!  
> > AFAIU yes, it's state, but not one that Guix can simply do away
> > with. 
> 
> It's easily possible to recreate /etc/passwd from scratch if the uids
> are
> always specified in <user-account>s and thus /etc/passwd would not
> need to
> be persistent state anymore.  Right now everything from /etc/passwd
> except
> the uid and the comment is already specified in <user-account>.
> 
> So Guix can indeed simply do away with the persistent state of
> /etc/passwd--that's why I suggested specifying the uids in the first
> place.
> 
> (By now I don't think that's the best way to make UIDs stable, but
> it's
> factually incorrect to assert that Guix can't simply do away with
> that
> persistent state specifically.  It can.)
> 
> > There is not yet a syntax for keeping secrets, which would be
> > needed to
> > fully populate /etc from config.scm.  Perhaps we'll get there some
> > day.
> 
> /etc/passwd does not contain secrets.  Neither does /etc/group.
> 
> And /etc/shadow doesn't contain uids.
> 
> So there is no conflict.
Point taken, it is indeed possibly to do away with one of those files,
but looking at them as a trio (as one ought to imo), I don't think
removing one while keeping the other(s) is the way to go.

Also if you do go that route, you would need a way to specify that your
passwd has hitherto been different to all other Guix installations;
hence forcing you to make system account [GU]IDs configurable once
again.

Regards,
Leo

bug#45571: Support stable uids and gids for all accounts

[Ludovic Courtès]

Hi Danny,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> It's easily possible to recreate /etc/passwd from scratch if the uids are
> always specified in <user-account>s and thus /etc/passwd would not need to
> be persistent state anymore.  Right now everything from /etc/passwd except
> the uid and the comment is already specified in <user-account>.
>
> So Guix can indeed simply do away with the persistent state of
> /etc/passwd--that's why I suggested specifying the uids in the first place.

I don’t think it can: UID/GID allocation is stateful by nature because
you don’t want to reuse a recently-used ID right away.  See the
allocation strategy in (gnu build accounts).

Ludo’.