Certificates in pure and containerized environments

Certificates in pure and containerized environments

[Konrad Hinsen]

Dear Guix experts,

I am trying to move the execution of a Python script into a pure
environment, but it fails because of certificate issues for which I am
looking for a good fix.

The minimal example I came up with is:

   guix environment --pure \
           --ad-hoc python -- \
           python3 -c 'import urllib.request; print(urllib.request.urlopen("http://wwwbis.sidc.be/DATA/uset/Wlight/2003/11/UPH20031109112104.FTS"))'

This fails with the error message

    urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1108)>

I have no idea why Python's urllib checks a certificate chain for a http
URL, but that's not my subject now. A variant that works fine is

   guix environment --pure --preserve=^SSL_CERT \
           --ad-hoc python -- \
           python3 -c 'import urllib.request; print(urllib.request.urlopen("http://wwwbis.sidc.be/DATA/uset/Wlight/2003/11/UPH20031109112104.FTS"))'

which adds two environment variables to the temporary environment:

   SSL_CERT_DIR=/home/hinsen/.guix-profile/etc/ssl/certs
   SSL_CERT_FILE=/home/hinsen/.guix-profile/etc/ssl/certs/ca-certificates.crt

Since all the certificates come from Guix, I expect that I should also
be able to use

   guix environment --pure \
           --ad-hoc python nss-certs -- \
           python3 -c 'import urllib.request; print(urllib.request.urlopen("http://wwwbis.sidc.be/DATA/uset/Wlight/2003/11/UPH20031109112104.FTS"))'

but this doesn't work - same error as initially.

I am not happy with exposing an environment variable from my personal
account, as I want my run to be reproducible and portable to any machine
running Guix, Ideally, I'd even move on to a containerized environment.
Is there any way to achieve this?

Thanks in advance,
  Konrad

Re: Certificates in pure and containerized environments

[Wiktor Żelazny]

[-- Attachment #1: Type: text/plain, Size: 752 bytes --]

On Thu, Sep 30, 2021 at 12:08:53PM +0200, Konrad Hinsen wrote:

>    guix environment --pure \
>            --ad-hoc python nss-certs -- \
>            python3 -c 'import urllib.request; print(urllib.request.urlopen("http://wwwbis.sidc.be/DATA/uset/Wlight/2003/11/UPH20031109112104.FTS"))'
>
> but this doesn't work - same error as initially.

Hi Konrad,

For some reason, it works for me with

   --ad-hoc python nss-certs guix -- \

. I’m neither sure if this is going to work on all machines (it works
when isolated with

   guix environment -C -N

, so there’s some hope), nor whether this solution is acceptable to you.
Perhaps, it’s abusing Guix. Maybe somebody more knowledgeable will
comment on this.

Bye,

WŻ

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: Certificates in pure and containerized environments

[zimoun]

Hi Konrad and Wiktór,

On Sun, 03 Oct 2021 at 18:45, Wiktor Żelazny <wz@freeshell.de> wrote:
> On Thu, Sep 30, 2021 at 12:08:53PM +0200, Konrad Hinsen wrote:
>
>>    guix environment --pure \
>>            --ad-hoc python nss-certs -- \
>>            python3 -c 'import urllib.request; print(urllib.request.urlopen("http://wwwbis.sidc.be/DATA/uset/Wlight/2003/11/UPH20031109112104.FTS"))'
>>
>> but this doesn't work - same error as initially.

Yeah for some reasom SSL_CERT_DIR is not exported…

> For some reason, it works for me with
>
>    --ad-hoc python nss-certs guix -- \

…but exported here.

--8<---------------cut here---------------start------------->8---
$ guix environment --ad-hoc python nss-certs
$ cat $GUIX_ENVIRONMENT/etc/profile
# Source this file to define all the relevant environment variables in Bash
# for this profile.  You may want to define the 'GUIX_PROFILE' environment
# variable to point to the "visible" name of the profile, like this:
#
#  GUIX_PROFILE=/path/to/profile ; \
#  source /path/to/profile/etc/profile
#
# When GUIX_PROFILE is undefined, the various environment variables refer
# to this specific profile generation.

export PATH="${GUIX_PROFILE:-/gnu/store/bp1xirq9p5cw36nkgi1131knhmhdzcvf-profile}/bin${PATH:+:}$PATH"
export PYTHONPATH="${GUIX_PROFILE:-/gnu/store/bp1xirq9p5cw36nkgi1131knhmhdzcvf-profile}/lib/python3.8/site-packages${PYTHONPATH:+:}$PYTHONPATH"
--8<---------------cut here---------------end--------------->8---

If any package depending on nss-certs is added, then it works, I guess.
For instance, a package totally unrelated, say r-reqon:

--8<---------------cut here---------------start------------->8---
$ guix environment --ad-hoc python nss-certs r-reqon
$ cat $GUIX_ENVIRONMENT/etc/profile
# Source this file to define all the relevant environment variables in Bash
# for this profile.  You may want to define the 'GUIX_PROFILE' environment
# variable to point to the "visible" name of the profile, like this:
#
#  GUIX_PROFILE=/path/to/profile ; \
#  source /path/to/profile/etc/profile
#
# When GUIX_PROFILE is undefined, the various environment variables refer
# to this specific profile generation.

export PATH="${GUIX_PROFILE:-/gnu/store/mj821vsw16c8krqm2c4syg2mdfzqy3j0-profile}/bin${PATH:+:}$PATH"
export CURL_CA_BUNDLE="${GUIX_PROFILE:-/gnu/store/mj821vsw16c8krqm2c4syg2mdfzqy3j0-profile}/etc/ssl/certs/ca-certificates.crt"
export SSL_CERT_FILE="${GUIX_PROFILE:-/gnu/store/mj821vsw16c8krqm2c4syg2mdfzqy3j0-profile}/etc/ssl/certs/ca-certificates.crt"
export SSL_CERT_DIR="${GUIX_PROFILE:-/gnu/store/mj821vsw16c8krqm2c4syg2mdfzqy3j0-profile}/etc/ssl/certs"
export PYTHONPATH="${GUIX_PROFILE:-/gnu/store/mj821vsw16c8krqm2c4syg2mdfzqy3j0-profile}/lib/python3.8/site-packages${PYTHONPATH:+:}$PYTHONPATH"
--8<---------------cut here---------------end--------------->8---


Hope that helps,
simon

Re: Certificates in pure and containerized environments

[Konrad Hinsen]

Hi Wiktór and Simon,

thanks for shedding some light on this strange behavior. After some more
exploration, the fundamental issue seems to be that many packages use
certificates but only a very small number declare a dependence on
nss-certs. In fact, nss-certs has only three direct dependents (icedtea,
ldns, and pypy) and 115 additional indirect dependents.  That includes
r-reqon from Simon's example, which depends on icedtea via r-rjava and
openjdk.

A radical fix would be to make openssl dependent on nss-certs. But
openssl really depends on the availability of some collection of
certificates, not on any specific one. Nor do icedtea, ldns, or pypy.

Some packages (e.g. openssl or curl) have a `native-search-paths`
declaration that also seems to have the desired effect. The following
environment contains SSL_CERT_DIR as well:

   guix environment --pure --ad-hoc python nss-certs openssl

Python actually lists openssl as a dependency, but that is apparently
not sufficient to propagate the environment variables.

Anyway, this looks like the best workaround for me for now: adding
openssl to my environment. It adds no software package to my
environment, only environment variables and an executable on $PATH.

Thanks again,
  Konrad

Re: Certificates in pure and containerized environments

[zimoun]

Hi Konrad,

On Mon, 04 Oct 2021 at 11:37, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:

> thanks for shedding some light on this strange behavior. After some more
> exploration, the fundamental issue seems to be that many packages use
> certificates but only a very small number declare a dependence on
> nss-certs. In fact, nss-certs has only three direct dependents (icedtea,
> ldns, and pypy) and 115 additional indirect dependents.  That includes
> r-reqon from Simon's example, which depends on icedtea via r-rjava and
> openjdk.

Yeah, I found it using “guix graph -t reverse-package nss-certs”. :-)


> Python actually lists openssl as a dependency, but that is apparently
> not sufficient to propagate the environment variables.

I think it is the issue: correctly propagate the environment variables.
But I do not know how.

Cheers,
simon

Re: Certificates in pure and containerized environments

[Maxim Cournoyer]

Hi,

Wiktor Żelazny <wz@freeshell.de> writes:

> On Thu, Sep 30, 2021 at 12:08:53PM +0200, Konrad Hinsen wrote:
>
>>    guix environment --pure \
>>            --ad-hoc python nss-certs -- \
>>            python3 -c 'import urllib.request; print(urllib.request.urlopen("http://wwwbis.sidc.be/DATA/uset/Wlight/2003/11/UPH20031109112104.FTS"))'
>>
>> but this doesn't work - same error as initially.
>
> Hi Konrad,
>
> For some reason, it works for me with
>
>    --ad-hoc python nss-certs guix -- \
>
> . I’m neither sure if this is going to work on all machines (it works
> when isolated with
>
>    guix environment -C -N
>
> , so there’s some hope), nor whether this solution is acceptable to you.
> Perhaps, it’s abusing Guix. Maybe somebody more knowledgeable will
> comment on this.

The key thing here is whether the certs are required by OpenSSL vs
GnuTLS.  The former honors SSL_CERT_DIR, while the later does not (I
opened an issue because I think it'd be nice to have them both honor it
the same here: [0]).  GnuTLS on Guix gets its certifications from the
hard coded location /etc/ssl/certs/.  This need to be bound in the
container; on a Guix System, it's also not enough to simply pass
/etc/ssl/certs/ as is, as these are symlinks to the store; you must also
expose the store or bind the etc/ssl/certs/ directory of the nss-certs
package directly.

I hope that helps!

Maxim

[0]  https://issues.guix.gnu.org/46779

Re: Certificates in pure and containerized environments

[Konrad Hinsen]

Hi Maxim,

> The key thing here is whether the certs are required by OpenSSL vs
> GnuTLS.  The former honors SSL_CERT_DIR, while the later does not (I
...

> I hope that helps!

Thanks, that certainly helps to understand the issues.

My preferred approach would be to manage all certificates as Guix
packages, and not have any environment variables.  That would be the
opposite of your proposal to make GnuTLS honor SSL_CERT_DIRS. It's
always a mess to have multiple uncoordinated environment managers.

I do see the difficulty for those who need personal certificates and
don't know how to package them in Guix, but that could be solved by a
dedicated tool.

Cheers,
  Konrad

Re: Certificates in pure and containerized environments

[Wiktor Żelazny]

[-- Attachment #1: Type: text/plain, Size: 503 bytes --]

On Mon, Oct 04, 2021 at 09:25:13AM +0200, zimoun wrote:
> On Sun, 03 Oct 2021 at 18:45, Wiktor Żelazny <wz@freeshell.de> wrote:
>
> > For some reason, it works for me with
> >
> >    --ad-hoc python nss-certs guix -- \

> If any package depending on nss-certs is added, then it works,

Would it make sense to define a minimal package that includes only
nss-certs input and use it for this purpose, then? A package that
inherits from "hello" with nss-certs input added, for instance?

WŻ

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: Certificates in pure and containerized environments

[Konrad Hinsen]

Wiktor Żelazny <wz@freeshell.de> writes:

> Would it make sense to define a minimal package that includes only
> nss-certs input and use it for this purpose, then? A package that
> inherits from "hello" with nss-certs input added, for instance?

That would be a quick fix. Could it become an obstacle to a better
long-term solution? Otherwise I like the idea.

Cheers,
  Konrad

Re: Certificates in pure and containerized environments

[Maxim Cournoyer]

Hello Konrad,

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> Hi Maxim,
>
>> The key thing here is whether the certs are required by OpenSSL vs
>> GnuTLS.  The former honors SSL_CERT_DIR, while the later does not (I
> ...
>
>> I hope that helps!
>
> Thanks, that certainly helps to understand the issues.
>
> My preferred approach would be to manage all certificates as Guix
> packages, and not have any environment variables.  That would be the
> opposite of your proposal to make GnuTLS honor SSL_CERT_DIRS. It's
> always a mess to have multiple uncoordinated environment managers.

I agree that managing certs with Guix has many benefits, and having
GnuTLS honor an SSL_CERTS_DIRS environment variable would enable that.
Remember that installing nss-certs or your certs of choice package to a
profile is not enough to have them discovered; something such as en
environment variable and a search path specification is also necessary.

Currently, even if you package you certs with Guix, if you install them
to a profile GnuTLS wouldn't know to use them unless you make them
available from /etc/ssl/certs/.

I hope that clarifies things.

Thanks,

Maxim

Re: Certificates in pure and containerized environments

[Konrad Hinsen]

Hi Maxim,

> I agree that managing certs with Guix has many benefits, and having
> GnuTLS honor an SSL_CERTS_DIRS environment variable would enable that.

Yes, but it would also make it hard to avoid non-Guix-managed
certificates from being used, be it through user (configuration)
error or malice.

> Remember that installing nss-certs or your certs of choice package to a
> profile is not enough to have them discovered; something such as en
> environment variable and a search path specification is also necessary.

That's exactly what I'd like to change (ideally, I don't know if that is
even possible). Configuration via Guix and nothing but
Guix. The only certificates being used are those defined in a Guix
profile or environment.

Cheers,
  Konrad.