Supporting sssd, preparing for nscd sunset

Supporting sssd, preparing for nscd sunset

[Ludovic Courtès]

Hello Guix!

Distros are increasingly relying on sssd, in particular Fedora and
derivatives, as a replacement for nscd, which is either unavailable or
deprecated.  The documented interface of Guix binaries to the host’s
name service switch (NSS) is currently nscd:

  https://guix.gnu.org/manual/en/html_node/Application-Setup.html#Name-Service-Switch

So it sounds like we’ll have to do something about it.

In <https://lists.gnu.org/archive/html/guix-devel/2020-08/msg00168.html>
I showed a workaround in cases where you cannot install nscd on the
machine, but we’ll need something that works out of the box.

Fellow NixOS hackers have been developing an nscd replacement
(implementing the same protocol, just not caching lookups):

  https://alternativebit.fr/posts/nixos/nsncd-some-updates/

The solution would be interesting for use on Guix System if and when
nscd is removed from glibc, but I don’t see it helpful on other distros,
at least not until/unless it becomes widespread.  (We can’t really ship
ourselves because it has to be linked against the host libc.)

For the record, glibc maintainer Carlos O’Donell brought up our use case
a while back on the glibc mailing list but it wasn’t particularly well
received:

  https://sourceware.org/pipermail/libc-alpha/2022-February/136741.html

I’m not sure where to go from here.  Long-term we probably need to
ensure a (non-caching) nscd remains part of glibc, and that glibc’s
client stubs can still talk that protocol; or perhaps nsncd will
gradually replace nscd, which is fine too.  If sssd becomes dominant, we
might just as well arrange to have NSS always load libnss_sss.so.

Thoughts?

Ludo’.

Re: Supporting sssd, preparing for nscd sunset

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi,

On Fri, Feb 23 2024, Ludovic Courtès wrote:

> The solution would be interesting for use on Guix System ... but I
> don’t see it helpful on other distros ... (We can’t really ship
> ourselves because it has to be linked against the host libc.)

To read more about nscd's significance on foreign distros, please see
our manual. [1] Carlos O'Donnell also referenced it in his messsage.

Kind regards
Felix

[1] https://guix.gnu.org/manual/en/html_node/Application-Setup.html#Name-Service-Switch-1

Re: Supporting sssd, preparing for nscd sunset

[Ricardo Wurmus]

Hi Ludo,

thanks for getting the discussion started.  This problem has been
weighing on me for the past months and I don't see a good way forward.

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> For the record, glibc maintainer Carlos O’Donell brought up our use case
> a while back on the glibc mailing list but it wasn’t particularly well
> received:
>
>   https://sourceware.org/pipermail/libc-alpha/2022-February/136741.html

DJ Delorie's response makes it sound like they would prefer that all of
the host system's NSS *configuration* (alongside the NSS modules) be
duplicated in the Guix "container", because that's not guaranteed to be
compatible.  That's not a way forward for us, because that would make
running Guix-built software much harder and require significant
configuration on the side of the user.

> [...] If sssd becomes dominant, we
> might just as well arrange to have NSS always load libnss_sss.so.

What are the other contenders here?  I assumed that sssd was what
everyone™ used nowadays.

At the MDC we've been using sssd on RHEL for a long time and we've seen
some problems with the naïve approach of just having the Guix System's
NSS load libnss_sss.so:

- In a Guix System VM we also had to explicitly set LD_PRELOAD or
  LD_LIBRARY_PATH in the current environment (not just nscd's
  environment) to make things like "id" work.  Having just NSS load the
  library was not enough to work with user accounts that are defined in
  LDAP, for example.

- A different Guix System VM was configured to use nss-pam-ldapd.  Upon
  reconfigure it attempted (and predictably failed) to delete all user
  accounts on LDAP, because none of them were defined locally.  I guess
  Guix System needs to be a little less eager to manage remote accounts
  that NSS plugins (including sssd) say exist.

- PAM in a container.

  While this is not strictly related to the disappearance of nscd, it is
  another obstacle relating to sssd.  On foreign distros I use Guix
  containers that bind-mount the host system's /var/lib/sss/pipes
  directory.  The hope was to use a native Guix build of libsss to talk
  to the socket at /var/lib/sss/pipes/pam (and potentially others) for
  authorizing users via the host system's sssd infrastructure.

  In "guix shell -C" this won't work, because sssd refuses to talk over
  that socket if the ownership is not root:root.  In the Guix shell
  container there is only ever *one* user account, so root-owned files
  are assigned to the overflow user account, and thus communication with
  the host system's pam socket is impossible.  I failed to fix this with
  subuids, because the "guix shell -C" process does not have sufficient
  privileges to map more than one user id.

I'll note that /var/lib/sss/pipes also contains an "nss" socket; I don't
know if it checks ownership on that socket, but it's possible that it
also cannot be used in "guix shell -C".

-- 
Ricardo

Re: Supporting sssd, preparing for nscd sunset

[Picnoir]

Hey Guix, Ludovic,

Some context: Flokli (CCd) and I are fighting the same issue on the
NixOS side.

There's two aspects in this migration story:

1. the daemon-side. IE. how to make sure a daemon is available on most
   distros to respond to the nscd socket
2. the client-side. IE. how to make sure we'll still have access to the
   glibc stubs sending queries to the daemon socket on the foreseeable
   future.

Let's talk about the daemon side first. As I was mentionning in the blog
post linked above, NixOS moved to Nsncd, a non caching re-implementation
of Nscd.

> I don’t see it helpful on other distros, at least not until/unless it
> becomes widespread. (We can’t really ship ourselves because it has to
> be linked against the host libc.)

Nsncd is already available in Debian[1] and Arch[2] (AUR). In reality,
some dowstreams, like Arch, are not distributing nscd anymore[3].
Meaning the only way to run Nix/Guix binaries on Arch without breaking
PAM is to go through Nsncd at the moment.

So, on the daemon side, on the long run, for us, the solution would be
to package Nsncd to Ubuntu and Fedora. We'd love to share that effort
with the Guix community.

Let's address the elephant in the room: Nsncd is written in Rust. I know
that language is not the most loved among the Guix crowd due to its
bootstrapping story. But looking at the amazing work Efraim put into the
toolchain, I don't think this should be a show stopper for Guix. We're
likely to get more Rust in core system parts in the future anyways.

Now, if you folks decide to part ways (sad), unscd[4] would likely be a
better starting point to implement a uncaching nscd daemon than the
current nscd implementation.

------

Client-side. There's two aspects to it:

1. keep the nscd stubs, hopefully without any patches in Glibc.
2. improve the current protocol.

For the stubs, let's take the worst-case scenario for us: upstream
decides to remove them. It seems like Carlos O'Donell[5] is up to
provide some guidance and support into hiding them behind a feature flag
and keep them in Glibc instead. So, even in our worst-case scenario, it
seems like we'll be okay-ish. We'll still need to put some people-hours
into it though.

From my perspective, the current protocol is not entirely satisfactory.
Implementing it in Nsncd was clearly no fun. But even besides this
aspect, it's currently not good enough for IPv6.

From the nscd protocol perspective, an IPv6 address is a serie of 16
octets. This is a fine representation for global and ULA addresses.
However, it's not good enough to fully qualify a link-local addresses:
you also need the scopeid (ie. interface id/interface name) that comes
with it .

This means that IPv6 local link-local addresses resolution is currently
broken for Guix/Nix programs. Such a resolution is sadly heavily used
in IPv6 mdns setups.

My point being: if we plan to keep using this nscd trick on the long run
(we currently do), and we end up doing a massive refactoring on the
Glibc side to hide the stubs, then it might be a good occasion to make
some improvements to the protocol itself at the same time. And who
knows, maybe we could make the bold move of redisiging altogether this
protocol.

For these protocol improvements, I definitely think our two communities
should sync and share notes.

Wow, that was long, sorry for the wall of text :)
Don't hesitate to Cc me and flokli for these Nscd discussions in the
future!

Picnoir

[1]: https://packages.debian.org/search?keywords=nsncd
[2]: https://aur.archlinux.org/packages/nsncd-git
[3]: https://gitlab.archlinux.org/archlinux/packaging/packages/glibc/-/blob/main/PKGBUILD?ref_type=heads#L53
[4]: https://github.com/bytedance/unscd
[5]: https://fosstodon.org/@codonell/111981707970893154

Re: Supporting sssd, preparing for nscd sunset

[Maxim Cournoyer]

Hi Ludovic,

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> Hello Guix!
>
> Distros are increasingly relying on sssd, in particular Fedora and
> derivatives, as a replacement for nscd, which is either unavailable or
> deprecated.  The documented interface of Guix binaries to the host’s
> name service switch (NSS) is currently nscd:
>
>   https://guix.gnu.org/manual/en/html_node/Application-Setup.html#Name-Service-Switch
>
> If sssd becomes dominant, we
> might just as well arrange to have NSS always load libnss_sss.so.
>
> Thoughts?

With nscd removed from glibc, can't we assume foreign distributions will
rely on sssd instead?  Do I understand correctly that the above
(configuring NSS to always load libnss_sss.so) would be a fine
replacement for our use of nscd?  If so, that seems the simplest/best
option to pursue to me.

-- 
Thanks,
Maxim

Re: Supporting sssd, preparing for nscd sunset

[Ludovic Courtès]

Hi Picnoir!

Picnoir <picnoir@alternativebit.fr> skribis:

> Nsncd is already available in Debian[1] and Arch[2] (AUR). In reality,
> some dowstreams, like Arch, are not distributing nscd anymore[3].
> Meaning the only way to run Nix/Guix binaries on Arch without breaking
> PAM is to go through Nsncd at the moment.
>
> So, on the daemon side, on the long run, for us, the solution would be
> to package Nsncd to Ubuntu and Fedora. We'd love to share that effort
> with the Guix community.

Excellent; I just saw Adam’s reply, let’s hope the Fedora bit can be
sorted out soon enough.  (I was afraid use of Rust will make packaging
harder, but apparently it’s not been a roadblock so far.)

Can you confirm nsncd can load and run popular NSS plugins like
nss-mdns and sss?

In terms of next actions for Guix, it means we can already update the
“Application Setup” section to recommend nsncd on platforms where nscd
is unavailable.

> Let's address the elephant in the room: Nsncd is written in Rust. I know
> that language is not the most loved among the Guix crowd due to its
> bootstrapping story. But looking at the amazing work Efraim put into the
> toolchain, I don't think this should be a show stopper for Guix. We're
> likely to get more Rust in core system parts in the future anyways.

Heheh.  Beyond preferences, most of us are also pragmatic at times :-).
If nsncd does the job, can be packaged without too much hassle, and can
be shared between Nix and Guix, then so be it!

> Client-side. There's two aspects to it:
>
> 1. keep the nscd stubs, hopefully without any patches in Glibc.
> 2. improve the current protocol.
>
> For the stubs, let's take the worst-case scenario for us: upstream
> decides to remove them. It seems like Carlos O'Donell[5] is up to
> provide some guidance and support into hiding them behind a feature flag
> and keep them in Glibc instead. So, even in our worst-case scenario, it
> seems like we'll be okay-ish. We'll still need to put some people-hours
> into it though.

Right.  We might need to resume the discussion on libc-alpha at some
point, to make sure our use case is known and understood.

> From my perspective, the current protocol is not entirely satisfactory.
> Implementing it in Nsncd was clearly no fun. But even besides this
> aspect, it's currently not good enough for IPv6.

One option that could also be considered, instead of changing the nscd
protocol, would be to have the glibc client stubs implement the sssd
protocol directly, given that sssd seems to have taken the role of
nscd-without-caching.

It’s certainly more work but a possibility to keep in mind while
discussing with upstream.

Thanks for your feedback!

Ludo’.

Re: Supporting sssd, preparing for nscd sunset

[Ludovic Courtès]

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Ludovic Courtès <ludovic.courtes@inria.fr> writes:
>
>> Hello Guix!
>>
>> Distros are increasingly relying on sssd, in particular Fedora and
>> derivatives, as a replacement for nscd, which is either unavailable or
>> deprecated.  The documented interface of Guix binaries to the host’s
>> name service switch (NSS) is currently nscd:
>>
>>   https://guix.gnu.org/manual/en/html_node/Application-Setup.html#Name-Service-Switch
>>
>> If sssd becomes dominant, we
>> might just as well arrange to have NSS always load libnss_sss.so.
>>
>> Thoughts?
>
> With nscd removed from glibc, can't we assume foreign distributions will
> rely on sssd instead?  Do I understand correctly that the above
> (configuring NSS to always load libnss_sss.so) would be a fine
> replacement for our use of nscd?  If so, that seems the simplest/best
> option to pursue to me.

It depends on what we mean by “configuring NSS to always load
libnss_sss.so”.

We could change the default NSS config, the one that’s used when
/etc/nsswitch.conf is missing, but that won’t help because in practice
distros do ship that file.

So what we would need to do is hack the NSS such that, when it sees
“nss_sss”, it kinda expands it to
/gnu/store/…-sssd-1.2.3/lib/libnss_sss.so.

That would complicate bootstrapping (as in: building ‘glibc-final’), but
may be worth considering.

Ludo’.

Re: Supporting sssd, preparing for nscd sunset

[Picnoir]

Hi Ludo, Guix,

> Can you confirm nsncd can load and run popular NSS plugins like
> nss-mdns and sss?

Nss-mdns works fine on IPv4. It won't work on IPv6 link-local addresses,
but that's due to the Nscd protocol issue I was talking about in the
previous email.

I did not test sss but I'd assume it to be working. Maybe Flokli did
test that?

The only known regression we're aware of is related to the Google
OSLogin PAM module, a module used by gcloud guests to retrieve the user
account accounts metadata. See
https://github.com/NixOS/nixpkgs/issues/218813. I **think** it comes
from the fact we're not "crasing" Nsncd properly when a PAM module is
segfaulting. I personally do not use google cloud, so I confess I did
dig too deep into that issue.

Aside from that, we're not aware of any other regressions. We migrated
from Nscd to Nsncd 2 NixOS stable versions ago, meaning Nsncd has been
already quite battletested in the wild by now.

> One option that could also be considered, instead of changing the nscd
> protocol, would be to have the glibc client stubs implement the sssd
> protocol directly, given that sssd seems to have taken the role of
> nscd-without-caching.
>
> It’s certainly more work but a possibility to keep in mind while
> discussing with upstream.

I had a look at the sssd protocol, it seems to be string-based, which
from my perspective is a better approach than the Nscd one. The protocol
also properly supports IPv6 link-local addresses scope ids.

As you said, it's quite some effort. I personally can't commit doing
such a re-implementation in the near future.

+1 about sending a summary to the glibc ML.

Picnoir