A corner case of broken reproducibility

A corner case of broken reproducibility

[Blake Shaw]

[-- Attachment #1: Type: text/plain, Size: 1238 bytes --]

Hiya Guix,

I imagine many folks are aware of this outcome, but from a quick search of
the archive I didnt find a discussion.

I decided to create a new user for building the lighter profile to deploy
in Singapore. Not knowing beyond the surface of how user profiles operate
at the Linux level, I changed my user name in my config without adding a
new user, thinking that only building a system with one profile will be
faster (with the servers having problems I'm having to fallback quite a
bit, so every bit of time saved helps)

But when I run $guix system roll-back, the account is broken. If I set the
password again as root and then try to log back in, it crashes during
loggin and in returned to the login menu. If I try to login with $sudo su
blake2b, I get the error: Permission denied: /home/blake2b/.bashrc. if I
remove  bashrc and then try to continue, the error happens again. And its
the same if I reconfigure with the new user.

So it seems this is a corner case of somewhat severe non-reproducibity (you
can't even access the environment). I'm guessing it has something to do
with setuid, perhaps the access rights are given to a strictly different
user even if the name is the same.

Is there any fix to this?

Ez,
Blake

[-- Attachment #2: Type: text/html, Size: 1571 bytes --]

Re: A corner case of broken reproducibility

[Felix Lechner]

Hi Blake,

On Tue, May 24, 2022 at 11:42 PM Blake Shaw <blake@sweatshoppe.org> wrote:
>
> I changed my user name in my config without adding a new user

I did that once in a new installation. In the second generation of my
configuration I changed the example user 'bob' to my own username. It
did not work well.

Due to precautions I was able to go back in as the root user and
adjust the permissions. The only unexpected part was that the symbolic
links for per-user profiles encode the relationship between user name
and uid.

Good luck and kind regards,
Felix Lechner

Re: A corner case of broken reproducibility

[Ludovic Courtès]

Hi,

Felix Lechner <felix.lechner@gmail.com> skribis:

> On Tue, May 24, 2022 at 11:42 PM Blake Shaw <blake@sweatshoppe.org> wrote:
>>
>> I changed my user name in my config without adding a new user
>
> I did that once in a new installation. In the second generation of my
> configuration I changed the example user 'bob' to my own username. It
> did not work well.
>
> Due to precautions I was able to go back in as the root user and
> adjust the permissions. The only unexpected part was that the symbolic
> links for per-user profiles encode the relationship between user name
> and uid.

Yes, this part is inherently stateful: Guix will not meddle with your
home directories and such.

Perhaps it should forcefully “chown -R” home directories at boot time,
so they have the right UID?  This has been discussed a few times,
especially in the context of system accounts; systemd-homed appears to
be doing exactly that.

Thoughts?

Ludo’.

Re: A corner case of broken reproducibility

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 2600 bytes --]

Ludovic Courtès schreef op ma 30-05-2022 om 17:58 [+0200]:
> Perhaps it should forcefully “chown -R” home directories at boot time,
> so they have the right UID?  This has been discussed a few times,

I haven't seen the "chown -R" suggestion yet in the context of homes
(only for system accounts, and at activation time, not only boot time).

> especially in the context of system accounts; systemd-homed appears to
> be doing exactly that.

I don't think the problem is that the uid of /home/... was wrong,
rather I think the problem is that Guix has forgotten the uid and hence
invents a new one to put in /etc/passwd instead of keeping the old one.

A pitfall (noticed in the context of system accounts): the user could
have created files outside /home (e.g. in /tmp).  IIUC, this would also
require a reboot to keep name<->uid consistent after "guix system
reconfigure".

A 'chown -R' doesn't seem great to me from a security perspective
(seems very easy to get something wrong, and the TOCTTOU-free chownat
hasn't been merged yet in Guile), a performance perspective (what if
you have a huge $HOME).  Also extra io -> slower boot + disk wear.
It also destroys some information, it's possible to intentionally have
files owned by other users inside $HOME.

Things that seem missing here to me:

  * a mechanism for remembering that an uid is still in use even though
    the user has been removed (previously mentioned solutions: keep the
    uid in /etc/passwd even though it is ‘removed’, or keep a separate
    /etc/passwd-graveyard or such, etc.).  For system accounts and user
    accounts.  Won't help in this particular case but would make more
    general adding/removing user accounts less fragile (avoid 
    accidental reuse).

  * a mechanism for telling Guix ‘I'm renaming the user account, not
    creating and removing a new one, so keep the uid’

  * some heuristics for detecting mistakes (e.g.: if Guix thinks it
    should create a directory /home/foo for uid 1234, but it notices
    there is already a directory /home/bar with that uid 1234, then
    that's super suspicious.  Likewise, if Guix thinks the home
    /home/foo should be owned by uid 1234, but it notices it's already
    owned by 1235!=1234, that's also suspicious).

    IIUC, Blake Shaw only changed their user name, not the home
    directory, so this would have detected the problem

  * some mechanism for resolving mistakes 

Or maybe something else entirely, whatever works without causing new
problems I guess.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: A corner case of broken reproducibility

[Ludovic Courtès]

Maxime Devos <maximedevos@telenet.be> skribis:

> I don't think the problem is that the uid of /home/... was wrong,
> rather I think the problem is that Guix has forgotten the uid and hence
> invents a new one to put in /etc/passwd instead of keeping the old one.
>
> A pitfall (noticed in the context of system accounts): the user could
> have created files outside /home (e.g. in /tmp).  IIUC, this would also
> require a reboot to keep name<->uid consistent after "guix system
> reconfigure".
>
> A 'chown -R' doesn't seem great to me from a security perspective
> (seems very easy to get something wrong, and the TOCTTOU-free chownat
> hasn't been merged yet in Guile), a performance perspective (what if
> you have a huge $HOME).  Also extra io -> slower boot + disk wear.
> It also destroys some information, it's possible to intentionally have
> files owned by other users inside $HOME.

There’s a talk by Lennart Poettering where he explains that, contrary to
what one might think, “chown -R $HOME” turns out to be fast enough that
systemd-homed can do that unconditionally (off the of my head).

> Things that seem missing here to me:
>
>   * a mechanism for remembering that an uid is still in use even though
>     the user has been removed (previously mentioned solutions: keep the
>     uid in /etc/passwd even though it is ‘removed’, or keep a separate
>     /etc/passwd-graveyard or such, etc.).  For system accounts and user
>     accounts.  Won't help in this particular case but would make more
>     general adding/removing user accounts less fragile (avoid 
>     accidental reuse).

How do you know that user “maxime” created today is “the same” as that
“maxime” deleted a while back?  You can’t.

(gnu build accounts) is stateful in that it makes sure UIDs aren’t
reused.  (This is roughly the same algorithm as used by Shadow.)

>   * a mechanism for telling Guix ‘I'm renaming the user account, not
>     creating and removing a new one, so keep the uid’

Every system generation stands alone though; it’s functional, stateless,
and all that.  What does “rename” mean in this context?

>   * some heuristics for detecting mistakes (e.g.: if Guix thinks it
>     should create a directory /home/foo for uid 1234, but it notices
>     there is already a directory /home/bar with that uid 1234, then
>     that's super suspicious.  Likewise, if Guix thinks the home
>     /home/foo should be owned by uid 1234, but it notices it's already
>     owned by 1235!=1234, that's also suspicious).

Yeah.

>   * some mechanism for resolving mistakes 

Sure.

Ludo’.

Re: A corner case of broken reproducibility

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 4185 bytes --]

Ludovic Courtès schreef op wo 01-06-2022 om 18:38 [+0200]:
> Things that seem missing here to me:
> 
>    * a mechanism for remembering that an uid is still in use even
> though
>      the user has been removed (previously mentioned solutions: keep
> the
>      uid in /etc/passwd even though it is ‘removed’, or keep a
> separate
>      /etc/passwd-graveyard or such, etc.).  For system accounts and
> user
>      accounts.  Won't help in this particular case but would make
> more
>      general adding/removing user accounts less fragile (avoid 
>      accidental reuse).

How do you know that user “maxime” created today is “the same” as that
“maxime” deleted a while back?  You can’t.

The point above was about remembering that uids (arbitrary unique
numbers) are still in use, and only tangentially about user names (the
/etc/passwd bit).

We can have a mechanism for registering that an uid that hasn't an
associated name anymore but might still be in use somewhere
(by a process or a file) and hence shouldn't be reused automatically,
without having to touch the concept of user names at all.

We don't have to know that they are the same or different (though we
could implement some kind of heuristic that old "foo" is the same as
new "foo" if that's desired), we only need some kind of mechanism to
stop automatically breaking things.

> (gnu build accounts) is stateful in that it makes sure UIDs aren’t
> reused.  (This is roughly the same algorithm as used by Shadow.)

It doesn't?  AFAICT it only takes /etc/passwd and /etc/groups in
account and there was some bug report reusing uids in system accounts
after removing a service (something about tor and gdm?), adding another
service and re-adding the original service or something like that.

> >    * a mechanism for telling Guix ‘I'm renaming the user account,
> > not
> >      creating and removing a new one, so keep the uid’
> 
> Every system generation stands alone though; it’s functional,
> stateless, and all that.  What does “rename” mean in this context?

A declaration in the configuration that in the past the user "bar" has
been named "foo", so if during reconfiguration Guix doesn't find "bar"
in /etc/passwd but it does find "foo", it shouldn't assign a new uid
and remove "foo" or such but rather change "foo:...:THIS-UID:.." entry
to "bar:...:THAT-UID":

(user-account
  (name "foo")
  (old-names '("bar" "baz"))
  (old-home-directories '("/home/bar" "/home/baz")) ; rename the old
/home/bar to /home/foo
  [...])

It's a bit state-y though in the sense that it refers to the previous
system configurations (and it wouldn't interact well with rollbacks
because the old configurations wouldn't have "foo" in old-names), so
some other solutions may be needed!

E.g., here's an alternative solution:

  * Require each user to be given unique identifier (UUID?) that cannot
    be changed (unlike uid or username or home directory).
    Record the UUID in /etc/passwd or such somehow.

    Example:

    + Configuration 1

    (user-account
      (name "foo")
      (uid 1234)
      (persistent-identifier "THE-UUID")
      (home-directory "/home/foo"))

    At first boot, this creates a foo:...:1234:...:THE-UUID entry in
    /etc/passwd as no user with THE-UUID exists yet.  Likewise, it also
    creates /home/foo

    + Configuration 2:

    (user-account
      (name "bar")
      (user 4321)
      (persistent-identifier "THE-UUID")        
      (home-directory "/home/bar"))

    At reconfiguration, Guix notices that THE-UUID already exists, so
    it renames the old home directory /home/foo (listed in
    /etc/passwd) to /home/bar, chowns /home/foo to 4321, and changes
    the passwd entry to use 4321 as uid and "bar" as username

    + Rolling-back to configuration 1: likewise.

A complication however: home directories on external media
that might not even be attached during reconfiguration, or
external media that are read-only.  And not being able to rename the
THE-UUID is a limitation.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: A corner case of broken reproducibility

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 1118 bytes --]

Ludovic Courtès schreef op wo 01-06-2022 om 18:38 [+0200]:
> There’s a talk by Lennart Poettering where he explains that, contrary to
> what one might think, “chown -R $HOME” turns out to be fast enough that
> systemd-homed can do that unconditionally (off the of my head).

Interesting.
Taking "find $HOME > /dev/null" as an approximation of how long "chown
-R $HOME" would take:

$ time find . > /dev/null 

real	0m7,066s
user	0m0,427s
sys	0m1,341s

Assuming that ‘unconditionally = only chown -R $HOME if the uid of
$HOME isn't what was expected’ (otherwise, +7sec for every boot),
that's not too bad.

That's on a SSD and not a hard disk though, I'd image it would be worse
on a hard disk.

Depending on the size of $HOME, it could be a lot longer though:

$ time find /gnu/store > /dev/null 

real	1m9,729s
user	0m4,135s
sys	0m13,840s

Might still be acceptable as long as uids aren't switched too often ...

Can we, as-is in Guix, assume we can modify /home/foo though?  E.g.
what if it's put on a separate storage medium not attached at boot.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: A corner case of broken reproducibility

[raingloom]

On Wed, 01 Jun 2022 22:09:11 +0200
Maxime Devos <maximedevos@telenet.be> wrote:

> Ludovic Courtès schreef op wo 01-06-2022 om 18:38 [+0200]:
> > There’s a talk by Lennart Poettering where he explains that,
> > contrary to what one might think, “chown -R $HOME” turns out to be
> > fast enough that systemd-homed can do that unconditionally (off the
> > of my head).  
> 
> Interesting.
> Taking "find $HOME > /dev/null" as an approximation of how long "chown
> -R $HOME" would take:
> 
> $ time find . > /dev/null 
> 
> real	0m7,066s
> user	0m0,427s
> sys	0m1,341s
> 
> Assuming that ‘unconditionally = only chown -R $HOME if the uid of
> $HOME isn't what was expected’ (otherwise, +7sec for every boot),
> that's not too bad.
> 
> That's on a SSD and not a hard disk though, I'd image it would be
> worse on a hard disk.
> 
> Depending on the size of $HOME, it could be a lot longer though:
> 
> $ time find /gnu/store > /dev/null 
> 
> real	1m9,729s
> user	0m4,135s
> sys	0m13,840s
> 
> Might still be acceptable as long as uids aren't switched too often
> ...
> 
> Can we, as-is in Guix, assume we can modify /home/foo though?  E.g.
> what if it's put on a separate storage medium not attached at boot.
> 
> Greetings,
> Maxime.

Could we instead check for existing homes and set uids in
/etc/passwd based on that instead? That's practically O(1), but is a
bit more involved.

Re: A corner case of broken reproducibility

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 572 bytes --]

raingloom schreef op wo 01-06-2022 om 22:41 [+0200]:
> Could we instead check for existing homes and set uids in
> /etc/passwd based on that instead? That's practically O(1), but is a
> bit more involved.

For this to work, the home directory may not be changed.  As Ludo wrote
(albeit about user names, not home directories):

> How do you know that user “maxime” created today is “the same” as
> that “maxime” deleted a while back?  You can’t.

Maybe it's an option, but there are other complications as well that
need to be resolved somehow.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: A corner case of broken reproducibility

[Ludovic Courtès]

Maxime Devos <maximedevos@telenet.be> skribis:

> Ludovic Courtès schreef op wo 01-06-2022 om 18:38 [+0200]:
>> There’s a talk by Lennart Poettering where he explains that, contrary to
>> what one might think, “chown -R $HOME” turns out to be fast enough that
>> systemd-homed can do that unconditionally (off the of my head).
>
> Interesting.

‘%gdm-activation’ has an optimization to recurse only if the top-level
directory has itself the wrong UID:

--8<---------------cut here---------------start------------->8---
(define %gdm-activation
  ;; Ensure /var/lib/gdm is owned by the "gdm" user.  This is normally the
  ;; case but could be wrong if the "gdm" user was created, then removed, and
  ;; then recreated under a different UID/GID: <https://bugs.gnu.org/37423>.
  (with-imported-modules '((guix build utils))
    #~(begin
        (use-modules (guix build utils))

        (let* ((gdm (getpwnam "gdm"))
               (uid (passwd:uid gdm))
               (gid (passwd:gid gdm))
               (st  (stat "/var/lib/gdm" #f)))
          ;; Recurse into /var/lib/gdm only if it has wrong ownership.
          (when (and st
                     (or (not (= uid (stat:uid st)))
                         (not (= gid (stat:gid st)))))
            (for-each (lambda (file)
                        (chown file uid gid))
                      (find-files "/var/lib/gdm"
                                  #:directories? #t)))))))
--8<---------------cut here---------------end--------------->8---

We can use that trick if needed: we’d get only one stat(2) call in the
common case.

Ludo’.

Re: A corner case of broken reproducibility

[Ludovic Courtès]

Hi,

Maxime Devos <maximedevos@telenet.be> skribis:

>> (gnu build accounts) is stateful in that it makes sure UIDs aren’t
>> reused.  (This is roughly the same algorithm as used by Shadow.)
>
> It doesn't?  AFAICT it only takes /etc/passwd and /etc/groups in
> account and there was some bug report reusing uids in system accounts
> after removing a service (something about tor and gdm?), adding another
> service and re-adding the original service or something like that.

I’m not sure what the conclusion of those bug reports were, but (gnu
build accounts) doesn’t reuse UIDs: you can see that in
‘user+group-databases’, which reads the initial /etc/{passwd,group}, and
passes them to ‘allocate-passwd’ and ‘allocate-group’, which “reserve”
those initial UIDs/GIDs.

This is exercised in tests/accounts.scm (search for “no reuse”).

There could be bugs of course, but in that case we should fix them.  :-)

Ludo’.

Re: A corner case of broken reproducibility

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 802 bytes --]

Ludovic Courtès schreef op do 02-06-2022 om 16:13 [+0200]:
> I’m not sure what the conclusion of those bug reports were, but (gnu
> build accounts) doesn’t reuse UIDs: you can see that in
> ‘user+group-databases’, which reads the initial /etc/{passwd,group}, and
> passes them to ‘allocate-passwd’ and ‘allocate-group’, which “reserve”
> those initial UIDs/GIDs.
> 
> This is exercised in tests/accounts.scm (search for “no reuse”).

Right, I guess I misremembered or misinterpreted the bug reports.
Or someone added and removed an account 900 times.

> There could be bugs of course, but in that case we should fix them.  :-)

So in theory there might still be a bug (or at least a limitation), albeit
unlikely to encounter in practice.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]