CA certificates

CA certificates

[Andreas Enge]

The attached patch series
1) adds a (private) python script to extract single certificates in .pem 
   format from a big textfile in mozilla source format;
2) adds the package nss-certs, which contains the certificates thus extracted
   in OUT/etc/ssl/certs, preprocessed with c_rehash for use with openssl;
3) adds "etc/ssl/certs" as a native-search-path for SSL_CERT_DIR to openssl.

So if you do a
   guix package -i openssl nss-certs youtube-dl
and add SSL_CERT_DIR as stipulated by the text output after the installation,
things work out of the box.

The search path definition means that we could have alternative root
certificate packages (potentially one per certification authority) and that
the user could install the ones he trusts.

The patches currently are in a branch wip-certs. Suggestions are welcome.

Andreas

Re: CA certificates

[Andreas Enge]

One more data point: fetchmail works out of the box with the certificates
and SSL_CERT_DIR. On the other hand, "git pull" on nixpkgs does not.

Andreas

Re: CA certificates

[Mark H Weaver]

Andreas Enge <andreas@enge.fr> writes:

> The attached patch series
> 1) adds a (private) python script to extract single certificates in .pem 
>    format from a big textfile in mozilla source format;
> 2) adds the package nss-certs, which contains the certificates thus extracted
>    in OUT/etc/ssl/certs, preprocessed with c_rehash for use with openssl;

Excellent, thanks very much! :)

> 3) adds "etc/ssl/certs" as a native-search-path for SSL_CERT_DIR to openssl.
>
> So if you do a
>    guix package -i openssl nss-certs youtube-dl
> and add SSL_CERT_DIR as stipulated by the text output after the installation,
> things work out of the box.
>
> The search path definition means that we could have alternative root
> certificate packages (potentially one per certification authority) and that
> the user could install the ones he trusts.

Sounds good!  It should be noted, however, that GnuTLS will currently
only use the certs in /etc/ssl/certs unless some application-specific
setting is provided.  This will later be improved with the 'p11-kit'
solution.

> The patches currently are in a branch wip-certs. Suggestions are
> welcome.

Regarding this commit:

> From b703198b70850017c2ed5e3510790898a214b7bd Mon Sep 17 00:00:00 2001
> From: Andreas Enge <andreas@enge.fr>
> Date: Tue, 10 Feb 2015 19:55:53 +0000
> Subject: gnu: Add nss-certs, certificates extracted from nss
> 
> * gnu/packages/certs.scm (nss-certs): New variable.
> ---
[...]
> +       #:phases
> +         (alist-cons-after
> +           'unpack 'install
> +           (lambda _
> +             (let ((certsdir (string-append %output "/etc/ssl/certs/")))
> +               (mkdir-p certsdir)
> +               (with-directory-excursion "nss/lib/ckfw/builtins/"
> +                 ;; extract single certificates from blob
> +                 (system* "certdata2pem.py" "certdata.txt")
> +                 ;; copy the .pem files into the output
> +                 (for-each
> +                   (lambda (file)
> +                     (copy-file file (string-append certsdir file)))
> +                   ;; FIXME: Some of the file names are UTF8 (?) and cause an
> +                   ;; error message such as 
> +                   ;; find-files:
> +                   ;; ./EBG_Elektronik_Sertifika_Hizmet_Sa??lay??c??s??:2.8.76.175.115.66.28.142.116.2.pem:
> +                   ;; No such file or directory
> +                   (find-files "." ".*\\.pem")))

Guile converts POSIX byte strings (e.g. file names) to strings using to
the current locale encoding, but the default locale in our build
environment is "C" which means ASCII-only.

I would advocate using a UTF-8 locale for all builds by default.

For now, I would try putting the following code at the beginning of your
custom 'install' phase:

--8<---------------cut here---------------start------------->8---
             (setenv "LOCPATH" (getcwd))
             (zero? (system* "localedef" "--no-archive"
                             "--prefix" (getcwd) "-i" "en_US"
                             "-f" "UTF-8" "./en_US.UTF-8"))
             (setlocale LC_ALL "en_US.UTF-8")
--8<---------------cut here---------------end--------------->8---

    Thanks!
      Mark

Re: CA certificates

[Andreas Enge]

On Thu, Feb 12, 2015 at 12:26:52PM -0500, Mark H Weaver wrote:
> Sounds good!  It should be noted, however, that GnuTLS will currently
> only use the certs in /etc/ssl/certs unless some application-specific
> setting is provided.  This will later be improved with the 'p11-kit'
> solution.

Indeed! I do not intend to work on it in the near future, so if someone
feels like it, please go ahead.

> Guile converts POSIX byte strings (e.g. file names) to strings using to
> the current locale encoding, but the default locale in our build
> environment is "C" which means ASCII-only.
> I would advocate using a UTF-8 locale for all builds by default.

I agree, this is the standard nowadays. And also because of the following
problem:

> For now, I would try putting the following code at the beginning of your
> custom 'install' phase:
> --8<---------------cut here---------------start------------->8---
>              (setenv "LOCPATH" (getcwd))
>              (zero? (system* "localedef" "--no-archive"
>                              "--prefix" (getcwd) "-i" "en_US"
>                              "-f" "UTF-8" "./en_US.UTF-8"))
>              (setlocale LC_ALL "en_US.UTF-8")
> --8<---------------cut here---------------end--------------->8---

It works, but ends with the following:
phase `install' succeeded after 8 seconds
@ build-succeeded /gnu/store/ryqpxy531n3njz04c3gvclzw2ljdxrbl-nss-certs-3.17.3.drv -
@ build-started /gnu/store/4adp88ayxq38r0zx5k4wy5lb8318jlx4-nss-certs-3.17.3.drv - x86_64-linux /usr/local/guix-git/var/log/guix/drvs/4a//dp88ayxq38r0zx5k4wy5lb8318jlx4-nss-certs-3.17.3.drv.bz2
Backtrace:
In ice-9/boot-9.scm:
...
ERROR: read error "/gnu/store/d2wasj07dhpqxwrgm99ssfjk2vrkgkcj-nss-certs-3.17.3/etc/ssl/certs/AC_Ra??z_Certic??mara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem" #f 2
grafting '/gnu/store/d2wasj07dhpqxwrgm99ssfjk2vrkgkcj-nss-certs-3.17.3' -> '/gnu/store/68b75w7phgdmd2h85gx1yrmx9f7mwg2m-nss-certs-3.17.3'...

So the build succeeds, but grafting the output does not. This should also
be done in an UTF-8 locale, I think.

Andreas

Locale of build environments

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> Guile converts POSIX byte strings (e.g. file names) to strings using to
> the current locale encoding, but the default locale in our build
> environment is "C" which means ASCII-only.
>
> I would advocate using a UTF-8 locale for all builds by default.

Note that this was already the case before locales were moved out of the
‘glibc’ package.  Also, the early bootstrap environment (until
‘glibc-final’ is used) doesn’t have locales.

For the non-bootstrap environment, we could make a ‘glibc-utf8-locale’
that would only include ‘en_US.UTF-8’ (or a few more, but which one?)
and add it to the implicit inputs of ‘gnu-build-system’.

WDYT?

Ludo’.

Re: Locale of build environments

[Andreas Enge]

On Thu, Feb 12, 2015 at 09:20:04PM +0100, Ludovic Courtès wrote:
> For the non-bootstrap environment, we could make a ‘glibc-utf8-locale’
> that would only include ‘en_US.UTF-8’ (or a few more, but which one?)
> and add it to the implicit inputs of ‘gnu-build-system’.

From what I can see, any one would be enough to work with packages that
have UTF-8 file names.

Andreas

Re: Locale of build environments

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:

> Mark H Weaver <mhw@netris.org> skribis:
>
>> Guile converts POSIX byte strings (e.g. file names) to strings using to
>> the current locale encoding, but the default locale in our build
>> environment is "C" which means ASCII-only.
>>
>> I would advocate using a UTF-8 locale for all builds by default.
>
> Note that this was already the case before locales were moved out of the
> ‘glibc’ package.  Also, the early bootstrap environment (until
> ‘glibc-final’ is used) doesn’t have locales.
>
> For the non-bootstrap environment, we could make a ‘glibc-utf8-locale’
> that would only include ‘en_US.UTF-8’ (or a few more, but which one?)
> and add it to the implicit inputs of ‘gnu-build-system’.
>
> WDYT?

Sounds good to me!

    Thanks,
      Mark

Re: Locale of build environments

[Ludovic Courtès]

Commit 5335c56 adds an ‘install-locale’ phase to gnu-build-system and
the next one adds glibc-utf8-locale to the implicit inputs.

During bootstrap, locales aren’t available so we’re still running on the
C locale, which is fine in practice because we only get to deal with
ASCII file names at that point and do not need UTF-8 for tests or
anything.

Commit 9cca706 adjust patch-and-repack to use UTF-8 as well when
possible–i.e., not during bootstrap.

The remain thing that needs to be done is ‘graft-derivation’.

Note to Mark: I didn’t use the #:env-vars approach we discussed on IRC,
where we’d pass GUILE_INSTALL_LOCALE=1 and LOCPATH=... directly because
that was trickier to do and didn’t add anything.

Ludo’.

Re: Locale of build environments

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:

> Commit 5335c56 adds an ‘install-locale’ phase to gnu-build-system and
> the next one adds glibc-utf8-locale to the implicit inputs.
>
> During bootstrap, locales aren’t available so we’re still running on the
> C locale, which is fine in practice because we only get to deal with
> ASCII file names at that point and do not need UTF-8 for tests or
> anything.
>
> Commit 9cca706 adjust patch-and-repack to use UTF-8 as well when
> possible–i.e., not during bootstrap.

Thanks!

However, there's a missing piece: I think we should set LANG to the
chosen locale, otherwise the top-level guile builder will be the only
process that uses UTF-8.  What do you think?

> The remain thing that needs to be done is ‘graft-derivation’.

Okay, I'll work on it.  I also want to optimize grafting.

> Note to Mark: I didn’t use the #:env-vars approach we discussed on IRC,
> where we’d pass GUILE_INSTALL_LOCALE=1 and LOCPATH=... directly because
> that was trickier to do and didn’t add anything.

Okay, makes sense.

    Thanks!
      Mark

Re: Locale of build environments

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 894 bytes --]

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Commit 5335c56 adds an ‘install-locale’ phase to gnu-build-system and
>> the next one adds glibc-utf8-locale to the implicit inputs.
>>
>> During bootstrap, locales aren’t available so we’re still running on the
>> C locale, which is fine in practice because we only get to deal with
>> ASCII file names at that point and do not need UTF-8 for tests or
>> anything.
>>
>> Commit 9cca706 adjust patch-and-repack to use UTF-8 as well when
>> possible–i.e., not during bootstrap.
>
> Thanks!
>
> However, there's a missing piece: I think we should set LANG to the
> chosen locale, otherwise the top-level guile builder will be the only
> process that uses UTF-8.  What do you think?

Yes, why not.  It’s not strictly necessary but it’s probably a good
idea.  Like this?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 2470 bytes --]

diff --git a/guix/build/gnu-build-system.scm b/guix/build/gnu-build-system.scm
index c3cc3ce..a2bd9d4 100644
--- a/guix/build/gnu-build-system.scm
+++ b/guix/build/gnu-build-system.scm
@@ -106,8 +106,12 @@ chance to be set."
   (catch 'system-error
     (lambda ()
       (setlocale locale-category locale)
-      (format (current-error-port) "using '~a' locale for category ~a~%"
-              locale locale-category)
+
+      ;; While we're at it, pass it to sub-processes.
+      (setenv (locale-category->string locale-category) locale)
+
+      (format (current-error-port) "using '~a' locale for category ~s~%"
+              locale (locale-category->string locale-category))
       #t)
     (lambda args
       ;; This is known to fail for instance in early bootstrap where locales
diff --git a/guix/build/utils.scm b/guix/build/utils.scm
index f24ed47..f43451b 100644
--- a/guix/build/utils.scm
+++ b/guix/build/utils.scm
@@ -21,6 +21,7 @@
 (define-module (guix build utils)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-11)
+  #:use-module (srfi srfi-60)
   #:use-module (ice-9 ftw)
   #:use-module (ice-9 match)
   #:use-module (ice-9 regex)
@@ -65,7 +66,9 @@
             patch-/usr/bin/file
             fold-port-matches
             remove-store-references
-            wrap-program))
+            wrap-program
+
+            locale-category->string))
 
 
 ;;;
@@ -909,6 +912,27 @@ the previous wrapper."
     (symlink wrapper prog-tmp)
     (rename-file prog-tmp prog)))
 
+\f
+;;;
+;;; Locales.
+;;;
+
+(define (locale-category->string category)
+  "Return the name of locale category CATEGORY, one of the 'LC_' constants.
+If CATEGORY is a bitwise or of several 'LC_' constants, an approximation is
+returned."
+  (letrec-syntax ((convert (syntax-rules ()
+                             ((_)
+                              (number->string category))
+                             ((_ first rest ...)
+                              (if (= first category)
+                                  (symbol->string 'first)
+                                  (convert rest ...))))))
+    (convert LC_ADDRESS LC_ALL LC_COLLATE LC_CTYPE
+             LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES LC_MONETARY
+             LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE
+             LC_TIME)))
+
 ;;; Local Variables:
 ;;; eval: (put 'call-with-output-file/atomic 'scheme-indent-function 1)
 ;;; eval: (put 'call-with-ascii-input-file 'scheme-indent-function 1)

[-- Attachment #3: Type: text/plain, Size: 191 bytes --]


>> The remain thing that needs to be done is ‘graft-derivation’.
>
> Okay, I'll work on it.  I also want to optimize grafting.

Excellent.

Thanks for your feedback!

Ludo’.

Re: Locale of build environments

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> Mark H Weaver <mhw@netris.org> skribis:
>
>> ludo@gnu.org (Ludovic Courtès) writes:
>>
>>> Commit 5335c56 adds an ‘install-locale’ phase to gnu-build-system and
>>> the next one adds glibc-utf8-locale to the implicit inputs.
>>>
>>> During bootstrap, locales aren’t available so we’re still running on the
>>> C locale, which is fine in practice because we only get to deal with
>>> ASCII file names at that point and do not need UTF-8 for tests or
>>> anything.
>>>
>>> Commit 9cca706 adjust patch-and-repack to use UTF-8 as well when
>>> possible–i.e., not during bootstrap.
>>
>> Thanks!
>>
>> However, there's a missing piece: I think we should set LANG to the
>> chosen locale, otherwise the top-level guile builder will be the only
>> process that uses UTF-8.  What do you think?
>
> Yes, why not.  It’s not strictly necessary but it’s probably a good
> idea.  Like this?

I went ahead with this change (to avoid an additional rebuild), but
comments are welcome of course.

Besides, commit e8c9f04 is interesting: ‘substitute*’ will now break
non-UTF-8 files by defaults (replacing invalid UTF-8 sequences with
question marks in the output.)  I don’t think this is a big issue, but
we’ll see in practice if it is.

Ludo’.

Re: Locale of build environments

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> Besides, commit e8c9f04 is interesting: ‘substitute*’ will now break
> non-UTF-8 files by defaults (replacing invalid UTF-8 sequences with
> question marks in the output.)

Based on that observation, commit dd0a8ef forced the ‘patch-*’
procedures to treat files as if they were ISO-8859-1–i.e., leaving their
byte sequence uninterpreted, and thus avoiding multibyte sequence
decoding errors.

Then, as Mark suggested, commit 4db8716 forces strict encoding/decoding
errors.

The problem then is that we’re getting things like
<http://hydra.gnu.org/build/263170/nixlog/1/raw>:

--8<---------------cut here---------------start------------->8---
phase `unpack' succeeded after 0 seconds
starting phase `patch-usr-bin-file'
patch-/usr/bin/file: ./configure: changing `/usr/bin/file' to `/gnu/store/a31g38iykai59jqmcwknxyjddc5zxm9b-file-5.22/bin/file'
patch-/usr/bin/file: ./configure: changing `/usr/bin/file' to `/gnu/store/a31g38iykai59jqmcwknxyjddc5zxm9b-file-5.22/bin/file'
patch-/usr/bin/file: ./configure: changing `/usr/bin/file' to `/gnu/store/a31g38iykai59jqmcwknxyjddc5zxm9b-file-5.22/bin/file'
patch-/usr/bin/file: ./configure: changing `/usr/bin/file' to `/gnu/store/a31g38iykai59jqmcwknxyjddc5zxm9b-file-5.22/bin/file'
patch-/usr/bin/file: ./configure: changing `/usr/bin/file' to `/gnu/store/a31g38iykai59jqmcwknxyjddc5zxm9b-file-5.22/bin/file'
patch-/usr/bin/file: ./configure: changing `/usr/bin/file' to `/gnu/store/a31g38iykai59jqmcwknxyjddc5zxm9b-file-5.22/bin/file'
patch-/usr/bin/file: ./configure: changing `/usr/bin/file' to `/gnu/store/a31g38iykai59jqmcwknxyjddc5zxm9b-file-5.22/bin/file'
patch-/usr/bin/file: ./configure: changing `/usr/bin/file' to `/gnu/store/a31g38iykai59jqmcwknxyjddc5zxm9b-file-5.22/bin/file'
patch-/usr/bin/file: ./configure: changing `/usr/bin/file' to `/gnu/store/a31g38iykai59jqmcwknxyjddc5zxm9b-file-5.22/bin/file'
Backtrace:

[...]

 745: 10 [patch-/usr/bin/file "./configure" #:file-command ...]
In ice-9/boot-9.scm:
 171: 9 [with-throw-handler #t ...]
 867: 8 [call-with-input-file "./configure" ...]
In /gnu/store/wcrp88qjv5bfhwcsxhbiqfh29da8pg81-module-import/guix/build/utils.scm:
 474: 7 [#<procedure 1998e80 at /gnu/store/wcrp88qjv5bfhwcsxhbiqfh29da8pg81-module-import/guix/build/utils.scm:473:10 (in)> #<input: ./configure 11>]
 500: 6 [#<procedure 1a092c0 at /gnu/store/wcrp88qjv5bfhwcsxhbiqfh29da8pg81-module-import/guix/build/utils.scm:496:6 (in out)> #<input: ./configure 11> ...]
In srfi/srfi-1.scm:
 465: 5 [fold #<procedure 17b41c0 at /gnu/store/wcrp88qjv5bfhwcsxhbiqfh29da8pg81-module-import/guix/build/utils.scm:500:32 (r+p line)> ...]
In /gnu/store/wcrp88qjv5bfhwcsxhbiqfh29da8pg81-module-import/guix/build/utils.scm:
 503: 4 [#<procedure 17b41c0 at /gnu/store/wcrp88qjv5bfhwcsxhbiqfh29da8pg81-module-import/guix/build/utils.scm:500:32 (r+p line)> # ...]
In ice-9/regex.scm:
 189: 3 [list-matches # ...]
 176: 2 [fold-matches # ...]
In unknown file:
   ?: 1 [regexp-exec # ...]
In ice-9/boot-9.scm:
 106: 0 [#<procedure 1998ec0 at ice-9/boot-9.scm:97:6 (thrown-k . args)> encoding-error ...]

ice-9/boot-9.scm:106:20: In procedure #<procedure 1998ec0 at ice-9/boot-9.scm:97:6 (thrown-k . args)>:
ice-9/boot-9.scm:106:20: Throw to key `encoding-error' with args `("scm_to_stringn" "cannot convert narrow string to output locale" 84 #f #f)'.
--8<---------------cut here---------------end--------------->8---

The failure here occurs when using ‘guile-final’ (which has full iconv
support.)  When it stumbles upon the © sign in ‘configure’, it reads it,
with ‘read-line’, as the sequence #\302 #\251.

However, when passing that line back to ‘regexp-exec’, ‘regex-exec’
calls ‘scm_to_locale_string’ on it, which fails with the error above:
this is because, in this build, we’re running on the C locale and #\302
aka. #\Â cannot be represented in ASCII (the encoding of the C locale.)

To solve that problem, commit 87c8b92 makes UTF-8 locales available
right after ‘guile-final’ is built.  That way, calls to
‘scm_to_locale_string’ actually convert to UTF-8, which always work.

(Note that the bootstrap Guile doesn’t have this problem because it uses
UTF-8 for everything and ignores locale settings.)

Hopefully we can enable full builds of ‘core-updates’ very soon now.

Ludo’.

Re: CA certificates

[Mark H Weaver]

Mark H Weaver <mhw@netris.org> writes:

> For now, I would try putting the following code at the beginning of your
> custom 'install' phase:
>
>              (setenv "LOCPATH" (getcwd))
>              (zero? (system* "localedef" "--no-archive"
>                              "--prefix" (getcwd) "-i" "en_US"
>                              "-f" "UTF-8" "./en_US.UTF-8"))
>              (setlocale LC_ALL "en_US.UTF-8")

I just realized that the 'zero?' is pointless in that position.  I would
simply omit it, since if it fails the 'setlocale' should raise an
exception anyway.

      Mark

Re: CA certificates

[Andreas Enge]

On Fri, Feb 13, 2015 at 02:28:10AM -0500, Mark H Weaver wrote:
> I just realized that the 'zero?' is pointless in that position.

Indeed. But the real problem, as I mentioned before, is that grafting fails
with a non-utf8 locale. We should switch to utf8 for everything.

Andreas

Re: CA certificates

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1008 bytes --]

Andreas Enge <andreas@enge.fr> skribis:

> The attached patch series
> 1) adds a (private) python script to extract single certificates in .pem 
>    format from a big textfile in mozilla source format;
> 2) adds the package nss-certs, which contains the certificates thus extracted
>    in OUT/etc/ssl/certs, preprocessed with c_rehash for use with openssl;
> 3) adds "etc/ssl/certs" as a native-search-path for SSL_CERT_DIR to openssl.

Cool.  I agree with Mark’s suggestion regarding UTF-8 file name
handling.  Other than that the patches LGTM.

All this X.509 stuff looks like a security quagmire but I suppose we’ll
have to live with it for some time more...

> So if you do a
>    guix package -i openssl nss-certs youtube-dl
> and add SSL_CERT_DIR as stipulated by the text output after the installation,
> things work out of the box.

Nice!  The (untested) patch below binds nss-certs to /etc/ssl/certs on
GuixSD, which should allow for more out-of-the-box goodness.  :-)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 883 bytes --]

diff --git a/gnu/system.scm b/gnu/system.scm
index 3fe7833..4b66e5d 100644
--- a/gnu/system.scm
+++ b/gnu/system.scm
@@ -41,6 +41,7 @@
   #:use-module (gnu packages man)
   #:use-module (gnu packages compression)
   #:use-module (gnu packages firmware)
+  #:use-module (gnu packages certs)
   #:autoload   (gnu packages cryptsetup) (cryptsetup)
   #:use-module (gnu services)
   #:use-module (gnu services dmd)
@@ -470,6 +471,7 @@ export ASPELL_CONF=\"dict-dir $HOME/.guix-profile/lib/aspell\"
                   ("shells" ,#~#$shells)
                   ("profile" ,#~#$profile)
                   ("hosts" ,#~#$hosts-file)
+                  ("ssl" ,#~(string-append #$nss-certs "/etc/ssl"))
                   ("localtime" ,#~(string-append #$tzdata "/share/zoneinfo/"
                                                  #$timezone))
                   ("sudoers" ,#~#$sudoers)))))

[-- Attachment #3: Type: text/plain, Size: 41 bytes --]


Thanks for working on it!

Ludo’.