[PATCH 0/6] WIP: nss: Update to 3.99

[PATCH 0/6] WIP: nss: Update to 3.99

[Christina O'Donnell]

Hi,

I've got as far as making nss 3.98 reproducible, however updating it to 3.99
results in 51 test failures. These are regressions, and worked correctly for
3.98. I'm not entirely sure what the issue is, but I've run out of time to
debug it this week, so I'm sending this patch up as is.

Up to patch 3 build correctly. Patch 4 is the first one that fails.

The issue specifically seems to all be related to FIPS:

    A PKCS #11 module returned CKR_DEVICE_ERROR, indicating that a problem has
    occurred with the token or slot.

If someone could take a look at this and see if there's anything I've missded
then I'd appreciate that. Otherwise I'm free to pick it back up again on
Tuesday.

Let me know if you have any questions.

Kind regards,
Christina

Christina O'Donnell (4):
  gnu: nss: Make reproducible.
  gnu: nss: Update to 3.99.
  gnu: nss-certs: Update to 3.99.
  WIP: nss: Attempting to resolve FIPS regression.

Zheng Junjie (2):
  gnu: nss: Fix cross-compilation.
  gnu: nspr: Fix cross-compilation.

 gnu/packages/certs.scm                        | 24 +++++--
 gnu/packages/nss.scm                          | 30 +++++++--
 .../patches/nss-Disable-library-signing.patch | 67 +++++++++++++++++++
 3 files changed, 111 insertions(+), 10 deletions(-)
 create mode 100644 gnu/packages/patches/nss-Disable-library-signing.patch


base-commit: 9a47ef6182b6a36354699efbdbedca17f24cd9b8
-- 
2.41.0

bug#40316: [PATCH 1/6] gnu: nss: Fix cross-compilation.

[Christina O'Donnell]

From: Zheng Junjie <zhengjunjie@iscas.ac.cn>

* gnu/packages/nss.scm (nss)[arguments]<#:make-flags>: When
cross-compilation, Add CROSS_COMPILE=1.
<#:phases>: When cross-compilation, Set env NATIVE_CC to gcc.

Change-Id: I5c9559a4b8cecf2cfc6c47d136d69c01a335faaf
Signed-off-by: Zheng Junjie <zhengjunjie@iscas.ac.cn>
---
 gnu/packages/nss.scm | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/gnu/packages/nss.scm b/gnu/packages/nss.scm
index 7e9ed49ead8..459e53bc1cf 100644
--- a/gnu/packages/nss.scm
+++ b/gnu/packages/nss.scm
@@ -154,6 +154,9 @@ (define-public nss
                                             (#$(target-linux?) "linux")
                                             (else ""))))
                        #~())
+                #$@(if (%current-target-system)
+                       #~("CROSS_COMPILE=1")
+                       #~())
                 (string-append "NSPR_INCLUDE_DIR="
                                (search-input-directory %build-inputs
                                                        "include/nspr"))
@@ -175,6 +178,10 @@ (define-public nss
             (lambda _
               (setenv "CC" #$(cc-for-target))
               (setenv "CCC" #$(cxx-for-target))
+              ;; TODO: Set this unconditionally
+              #$@(if (%current-target-system)
+                     #~((setenv "NATIVE_CC" "gcc"))
+                     #~())
               ;; No VSX on powerpc-linux.
               #$@(if (target-ppc32?)
                      #~((setenv "NSS_DISABLE_CRYPTO_VSX" "1"))
-- 
2.41.0

bug#40316: [PATCH 2/6] gnu: nspr: Fix cross-compilation.

[Christina O'Donnell]

From: Zheng Junjie <zhengjunjie@iscas.ac.cn>

* gnu/packages/nss.scm (nspr)[arguments]<#:configure-flags>: When
cross-compilation, Add HOST_CC=gcc.

Change-Id: I337f217f153f8cc3a713906643d6fab9115056e9
Signed-off-by: Zheng Junjie <zhengjunjie@iscas.ac.cn>
---
 gnu/packages/nss.scm | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/nss.scm b/gnu/packages/nss.scm
index 459e53bc1cf..0baafe2f373 100644
--- a/gnu/packages/nss.scm
+++ b/gnu/packages/nss.scm
@@ -71,7 +71,10 @@ (define-public nspr
       #~(list "--disable-static"
               "--enable-64bit"
               (string-append "LDFLAGS=-Wl,-rpath="
-                             (assoc-ref %outputs "out") "/lib"))
+                             (assoc-ref %outputs "out") "/lib")
+              #$@(if (%current-target-system)
+                     #~("HOST_CC=gcc")
+                     #~()))
       ;; Use fixed timestamps for reproducibility.
       #:make-flags #~'("SH_DATE='1970-01-01 00:00:01'"
                        ;; This is epoch 1 in microseconds.
-- 
2.41.0

[PATCH 3/6] gnu: nss: Make reproducible.

[Christina O'Donnell]

gnu/packages/patches/nss-Disable-library-signing.patch: Disable library
signing to make the build reproducible.
gnu/packages/nss.scm (nss): Apply this new patch.

Change-Id: I7860bae219ecc4a79423a590c27a1097ae2e7874
---
 gnu/packages/nss.scm                          |  3 +-
 .../patches/nss-Disable-library-signing.patch | 67 +++++++++++++++++++
 2 files changed, 69 insertions(+), 1 deletion(-)
 create mode 100644 gnu/packages/patches/nss-Disable-library-signing.patch

diff --git a/gnu/packages/nss.scm b/gnu/packages/nss.scm
index 0baafe2f373..b608a995577 100644
--- a/gnu/packages/nss.scm
+++ b/gnu/packages/nss.scm
@@ -124,7 +124,8 @@ (define-public nss
               ;; Create nss.pc and nss-config.
               (patches (search-patches "nss-3.56-pkgconfig.patch"
                                        "nss-getcwd-nonnull.patch"
-                                       "nss-increase-test-timeout.patch"))
+                                       "nss-increase-test-timeout.patch"
+                                       "nss-Disable-library-signing.patch"))
               (modules '((guix build utils)))
               (snippet
                '(begin
diff --git a/gnu/packages/patches/nss-Disable-library-signing.patch b/gnu/packages/patches/nss-Disable-library-signing.patch
new file mode 100644
index 00000000000..b488d29dcad
--- /dev/null
+++ b/gnu/packages/patches/nss-Disable-library-signing.patch
@@ -0,0 +1,67 @@
+From 4734b834755822f962af29e9395daa7338084e21 Mon Sep 17 00:00:00 2001
+Message-ID: <4734b834755822f962af29e9395daa7338084e21.1714059680.git.cdo@mutix.org>
+From: Christina O'Donnell <cdo@mutix.org>
+Date: Thu, 25 Apr 2024 16:35:50 +0100
+Subject: [PATCH] nss: Disable library signing.
+
+---
+ nss/cmd/shlibsign/Makefile | 32 +-------------------------------
+ 1 file changed, 1 insertion(+), 31 deletions(-)
+
+diff --git a/nss/cmd/shlibsign/Makefile b/nss/cmd/shlibsign/Makefile
+index a119205..7a85c1d 100644
+--- a/nss/cmd/shlibsign/Makefile
++++ b/nss/cmd/shlibsign/Makefile
+@@ -43,22 +43,9 @@ EXTRA_SHARED_LIBS += \
+ 
+ endif
+ 
+-
+-# sign any and all shared libraries that contain the word freebl
+-ifeq ($(NSS_BUILD_WITHOUT_SOFTOKEN),1)
++# Disable library signing as it's non-deterministic
+ CHECKLIBS =
+ CHECKLOC =
+-else
+-CHECKLIBS = $(DIST)/lib/$(DLL_PREFIX)softokn3.$(DLL_SUFFIX)
+-CHECKLIBS += $(wildcard $(DIST)/lib/$(DLL_PREFIX)freebl*3.$(DLL_SUFFIX))
+-ifndef NSS_DISABLE_DBM
+-CHECKLIBS += $(DIST)/lib/$(DLL_PREFIX)nssdbm3.$(DLL_SUFFIX)
+-endif
+-CHECKLOC = $(CHECKLIBS:.$(DLL_SUFFIX)=.chk)
+-
+-MD_LIB_RELEASE_FILES = $(CHECKLOC)
+-ALL_TRASH += $(CHECKLOC)
+-endif
+ 
+ #######################################################################
+ # (5) Execute "global" rules. (OPTIONAL)                              #
+@@ -78,23 +65,6 @@ include $(CORE_DEPTH)/coreconf/rules.mk
+ 
+ include ../platrules.mk
+ 
+-SRCDIR = $(call core_abspath,.)
+-
+-%.chk: %.$(DLL_SUFFIX) 
+-ifeq ($(OS_TARGET), OS2)
+-	cd $(OBJDIR) ; cmd.exe /c $(SRCDIR)/sign.cmd $(DIST) \
+-	$(call core_abspath,$(OBJDIR)) $(OS_TARGET) \
+-	$(call core_abspath,$(NSPR_LIB_DIR)) $(call core_abspath,$<)
+-else
+-    ifeq ($(CROSS_COMPILE),1)
+-	# do nothing
+-    else
+-	cd $(OBJDIR) ; sh $(SRCDIR)/sign.sh $(call core_abspath,$(DIST)) \
+-	$(call core_abspath,$(OBJDIR)) $(OS_TARGET) \
+-	$(call core_abspath,$(NSPR_LIB_DIR)) $(call core_abspath,$<)
+-    endif
+-endif
+-
+ libs: install
+ ifdef CHECKLOC
+ 	$(MAKE) $(CHECKLOC)
+
+base-commit: 2951778f8e8855bed24754a57ecc43f02a2843dd
+-- 
+2.41.0
+
-- 
2.41.0

[PATCH 4/6] gnu: nss: Update to 3.99.

[Christina O'Donnell]

gnu/packages/nss.scm (nss): Update to 3.99.

Change-Id: Iba6c9dc2956cc0febb62a1c471add899250fa489
---
 gnu/packages/nss.scm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/nss.scm b/gnu/packages/nss.scm
index b608a995577..80667d8affe 100644
--- a/gnu/packages/nss.scm
+++ b/gnu/packages/nss.scm
@@ -109,7 +109,7 @@ (define-public nss
     ;; IMPORTANT: Also update and test the nss-certs package, which duplicates
     ;; version and source to avoid a top-level variable reference & module
     ;; cycle.
-    (version "3.88.1")
+    (version "3.99")
     (source (origin
               (method url-fetch)
               (uri (let ((version-with-underscores
@@ -120,7 +120,7 @@ (define-public nss
                       "nss-" version ".tar.gz")))
               (sha256
                (base32
-                "15il9fsmixa1r4446zq1wl627sg0hz9h67w6kjxz273xz3nl7li7"))
+                "1g89ig40gfi1sp02gybvl2z818lawcnrqjzsws36cdva834c5maw"))
               ;; Create nss.pc and nss-config.
               (patches (search-patches "nss-3.56-pkgconfig.patch"
                                        "nss-getcwd-nonnull.patch"
-- 
2.41.0

[PATCH 5/6] gnu: nss-certs: Update to 3.99.

[Christina O'Donnell]

gnu/packages/certs.scm (nss-certs-3.88.1): New variable.
(nss-certs-3.98): Update and rename to nss-certs-3.99.
(nss-certs): Update to 3.99.

Change-Id: I2f5f737d44d08497d4f5e0e07557be36d2f1f070
---
 gnu/packages/certs.scm | 24 +++++++++++++++++++-----
 1 file changed, 19 insertions(+), 5 deletions(-)

diff --git a/gnu/packages/certs.scm b/gnu/packages/certs.scm
index 7078c7c8d11..7aa96493fbe 100644
--- a/gnu/packages/certs.scm
+++ b/gnu/packages/certs.scm
@@ -125,7 +125,7 @@ (define-public certdata2pem
 that was originally contributed to Debian.")
       (license license:isc))))
 
-(define-public nss-certs
+(define-public nss-certs-3.88.1
   (package
     (name "nss-certs")
     ;; XXX We used to refer to the nss package here, but that eventually caused
@@ -188,10 +188,10 @@ (define-public nss-certs
     (home-page "https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS")
     (license license:mpl2.0)))
 
-(define-public nss-certs-3.98
+(define-public nss-certs-3.99
   (package
-    (inherit nss-certs)
-    (version "3.98")
+    (inherit nss-certs-3.88.1)
+    (version "3.99")
     (source (origin
               (method url-fetch)
               (uri (let ((version-with-underscores
@@ -202,7 +202,21 @@ (define-public nss-certs-3.98
                       "nss-" version ".tar.gz")))
               (sha256
                (base32
-                "1kh98amfklrq6915n4mlbrcqghc3srm7rkzs9dkh21jwscrwqjgm"))))))
+                "15il9fsmixa1r4446zq1wl627sg0hz9h67w6kjxz273xz3nl7li7"))
+              ;; Create nss.pc and nss-config.
+              (patches (search-patches "nss-3.56-pkgconfig.patch"
+                                       "nss-getcwd-nonnull.patch"
+                                       "nss-increase-test-timeout.patch"
+                                       "nss-Disable-library-signing.patch"))
+              (modules '((guix build utils)))
+              (snippet
+               '(begin
+                  ;; Delete the bundled copy of these libraries.
+                  (delete-file-recursively "nss/lib/zlib")
+                  (delete-file-recursively "nss/lib/sqlite")))))))
+
+(define-public nss-certs
+  nss-certs-3.99)
 
 (define-public le-certs
   (package
-- 
2.41.0

[PATCH 6/6] WIP: nss: Attempting to resolve FIPS regression.

[Christina O'Donnell]

There are 51 new test failures which all appear to be related to FIPS.

For example:

modutil -dbdir /tmp/guix-build-nss-3.99.drv-0/nss-3.99/tests_results/security/localhost.1/fips -fips true

WARNING: Performing this operation while the browser is running could cause
corruption of your security databases. If the browser is currently running,
you should exit browser before continuing this operation. Type
'q <enter>' to abort, or <enter> to continue:
A PKCS #11 module returned CKR_DEVICE_ERROR, indicating that a problem has occurred with the token or slot.
ERROR: Unable to switch FIPS modes.
cert.sh: #291: Enable FIPS mode on database for FIPS PUB 140 Test Certificate (11)  - FAILED
cert.sh ERROR: Enable FIPS mode on database for FIPS PUB 140 Test Certificate failed 11

Change-Id: If0d57bb9e129eb862fae1a28d9779c6100e0a23d
---
 gnu/packages/nss.scm | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/nss.scm b/gnu/packages/nss.scm
index 80667d8affe..a8fb6965c2c 100644
--- a/gnu/packages/nss.scm
+++ b/gnu/packages/nss.scm
@@ -134,6 +134,10 @@ (define-public nss
                   (delete-file-recursively "nss/lib/sqlite")))))
     (build-system gnu-build-system)
     (outputs '("out" "bin"))
+    ;; (search-paths
+    ;;  (list (search-path-specification
+    ;;         (variable "LD_LIBRARY_PATH")
+    ;;         (files '("lib")))))
     (arguments
      (list
       #:make-flags
@@ -161,12 +165,15 @@ (define-public nss
                 #$@(if (%current-target-system)
                        #~("CROSS_COMPILE=1")
                        #~())
+                (string-append "NSS_FORCE_FIPS=1")
+                (string-append "NSPR_LIB_DIR="
+                               (string-append #$nspr "/lib"))
                 (string-append "NSPR_INCLUDE_DIR="
                                (search-input-directory %build-inputs
                                                        "include/nspr"))
                 ;; Add $out/lib/nss to RPATH.
                 (string-append "RPATH=" rpath)
-                (string-append "LDFLAGS=" rpath)))
+                (string-append "LDFLAGS=" rpath " -L" #$nspr "/lib")))
       #:modules '((guix build gnu-build-system)
                   (guix build utils)
                   (ice-9 ftw)
@@ -203,6 +210,8 @@ (define-public nss
                     (setenv "DOMSUF" "localdomain")
                     (setenv "USE_IP" "TRUE")
                     (setenv "IP_ADDRESS" "127.0.0.1")
+                    ;; (setenv "LD_LIBRARY_PATH"
+                    ;;         (string-append (getenv "LD_LIBRARY_PATH")))
 
                     ;; The "PayPalEE.cert" certificate expires every six months,
                     ;; leading to test failures:
-- 
2.41.0

Re: bug#40316: nss not reproducible

[Ludovic Courtès]

Hi Christina,

Nice work!

Christina O'Donnell <cdo@mutix.org> skribis:

> I've got as far as making nss 3.98 reproducible, however updating it to 3.99
> results in 51 test failures. These are regressions, and worked correctly for
> 3.98. I'm not entirely sure what the issue is, but I've run out of time to
> debug it this week, so I'm sending this patch up as is.

Not sure if this is related, but we’re seeing test failures due to
timing issues right now with 3.98:

  https://issues.guix.gnu.org/70693

Thank you!

Ludo’.

Re: bug#40316: nss not reproducible

[Christina O'Donnell]

Hi Ludo',

This doesn't look directly related. I haven't seen anything like this 
occur when I build it.

Tangentially, given how long nss takes to build, do you think that it'd 
be worth shaving it down to a single test pass? Currently it runs each 
test up to 3 times, which takes ~1h on my machine with no other build 
running. Running only the standard pass takes 2.5-3x less time, which is 
a huge quality of life improvement.

Kind regards,

Christina

On 02/05/2024 09:15, Ludovic Courtès wrote:
> Hi Christina,
>
> Nice work!
>
> Christina O'Donnell <cdo@mutix.org> skribis:
>
>> I've got as far as making nss 3.98 reproducible, however updating it to 3.99
>> results in 51 test failures. These are regressions, and worked correctly for
>> 3.98. I'm not entirely sure what the issue is, but I've run out of time to
>> debug it this week, so I'm sending this patch up as is.
> Not sure if this is related, but we’re seeing test failures due to
> timing issues right now with 3.98:
>
>    https://issues.guix.gnu.org/70693
>
> Thank you!
>
> Ludo’.

Re: bug#40316: nss not reproducible

[Ludovic Courtès]

Hi,

Christina O'Donnell <cdo@mutix.org> skribis:

> Tangentially, given how long nss takes to build, do you think that
> it'd be worth shaving it down to a single test pass? Currently it runs
> each test up to 3 times, which takes ~1h on my machine with no other
> build running. Running only the standard pass takes 2.5-3x less time,
> which is a huge quality of life improvement.

Currently we run ./nss/tests/all.sh, which I suppose is what upstream
recommends to run tests.

For sure I’d be happy if the test suite could run faster, but does
upstream offer such an option?  When you say “a single pass”, is that
something upstream supports?

Thanks,
Ludo’.

Re: bug#40316: nss not reproducible

[Christina O'Donnell]

Hi,

On 06/05/2024 11:12, Ludovic Courtès wrote:
> Hi,
>
> Christina O'Donnell <cdo@mutix.org> skribis:
>
>> Tangentially, given how long nss takes to build, do you think that
>> it'd be worth shaving it down to a single test pass? Currently it runs
>> each test up to 3 times, which takes ~1h on my machine with no other
>> build running. Running only the standard pass takes 2.5-3x less time,
>> which is a huge quality of life improvement.
> Currently we run ./nss/tests/all.sh, which I suppose is what upstream
> recommends to run tests.
>
> For sure I’d be happy if the test suite could run faster, but does
> upstream offer such an option?  When you say “a single pass”, is that
> something upstream supports?
Yes, you can control the tests by setting environment variables 
NSS_TESTS to a list of tests and NSS_CYCLES to a list of 'cycles' (what 
I previously called passes). The default is:

"standard pkix threadunsafe"

* 'standard' runs all of the below tests with default settings: "cipher 
lowhash cert dbtests tools sdr crmf smime ssl ocsp merge pkits ec gtests 
ssl_gtests policy"

* 'pkix' runs the tests "lowhash libpkix cert tools ssl ocsp pkits ec 
gtests ssl_gtests policy" with PKIX enabled.

* 'thread_unsafe' runs "ssl ssl_gtests" with "THREAD_UNSAFE" enabled.

My thinking would be to run the thread_unsafe cycle normally, but to 
reduce the test overlap between standard and pkix however, I can't say 
that I'm knowledgeable enough of NSS to claim that that wouldn't leave 
gaps that might bite us some point down the line. So it might be best to 
leave it as is unless someone familiar with NSS can confirm that it'd be 
safe to disable some tests/cycles.

Kind regards,

Christina

Re: bug#40316: nss not reproducible

[Ludovic Courtès]

Hi,

Christina O'Donnell <cdo@mutix.org> skribis:

> On 06/05/2024 11:12, Ludovic Courtès wrote:

[...]

>> For sure I’d be happy if the test suite could run faster, but does
>> upstream offer such an option?  When you say “a single pass”, is that
>> something upstream supports?
> Yes, you can control the tests by setting environment variables
> NSS_TESTS to a list of tests and NSS_CYCLES to a list of 'cycles'
> (what I previously called passes). The default is:
>
> "standard pkix threadunsafe"
>
> * 'standard' runs all of the below tests with default settings:
>   "cipher lowhash cert dbtests tools sdr crmf smime ssl ocsp merge
>   pkits ec gtests ssl_gtests policy"
>
> * 'pkix' runs the tests "lowhash libpkix cert tools ssl ocsp pkits ec
>   gtests ssl_gtests policy" with PKIX enabled.
>
> * 'thread_unsafe' runs "ssl ssl_gtests" with "THREAD_UNSAFE" enabled.

Interesting.

> My thinking would be to run the thread_unsafe cycle normally, but to
> reduce the test overlap between standard and pkix however, I can't say
> that I'm knowledgeable enough of NSS to claim that that wouldn't leave
> gaps that might bite us some point down the line. So it might be best
> to leave it as is unless someone familiar with NSS can confirm that
> it'd be safe to disable some tests/cycles.

Right, there doesn’t seem to be an obvious way to disable those without
also weakening test coverage.  I wonder what Debian and others are
doing.

Thanks for explaining!

Ludo’.