[bug#55067] [PATCH] gnu: nomad-optimizer: Disable failing test.

[bug#55067] [PATCH] gnu: nomad-optimizer: Disable failing test.

[Paul A. Patience]

* gnu/packages/maths.scm (nomad-optimizer)[arguments]: Disable the
examples/advanced/library/PSDMads test.
---
It turns out the PSDMads test fails randomly.
I've disabled it.

 gnu/packages/maths.scm | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index f4778179d6..26488a614c 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -2008,6 +2008,9 @@ (define-public nomad-optimizer
          (add-after 'fix-sources-for-build 'fix-sources-for-tests
            (lambda _
              (substitute* "examples/CMakeLists.txt"
+               ;; This test passes only sometimes.
+               (("^ +add_subdirectory\\(\\$\\{CMAKE_CURRENT_SOURCE_DIR\\}/advanced/library/PSDMads\\)\n")
+                "")
                ;; examples/basic/batch/example3 is accidentally omitted.
                (("^(add_subdirectory\\(\\$\\{CMAKE_CURRENT_SOURCE_DIR\\}/basic/batch/example)2(\\)\n)"
                  _ prefix suffix)
@@ -2052,10 +2055,6 @@ (define-public nomad-optimizer
               "examples/advanced/library/exampleSuggestAndObserve/cache0.txt")

              (let ((builddir (string-append (getcwd) "/../build")))
-               ;; For some reason, omitting this fix causes the
-               ;; examples/advanced/library/PSDMads test to fail, even though
-               ;; it doesn't seem to reference any part of the FixedVariable
-               ;; test.
                (let ((dir "examples/advanced/library/FixedVariable"))
                  (substitute* (string-append dir "/fixedVariable.cpp")
                    (("^( +std::string sExe = ).*" _ prefix)
--
2.35.1

[bug#55067] [PATCH v2] gnu: nomad-optimizer: Disable failing test.

[Paul A. Patience]

* gnu/packages/maths.scm (nomad-optimizer)[arguments]: Disable the
examples/advanced/library/PSDMads test.
---
Just updating the copyright header too.

 gnu/packages/maths.scm | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index f4778179d6..190b62fca1 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -45,7 +45,7 @@
 ;;; Copyright © 2021 Gerd Heber <gerd.heber@gmail.com>
 ;;; Copyright © 2021 Franck Pérignon <franck.perignon@univ-grenoble-alpes.fr>
 ;;; Copyright © 2021 Philip McGrath <philip@philipmcgrath.com>
-;;; Copyright © 2021 Paul A. Patience <paul@apatience.com>
+;;; Copyright © 2021-2022 Paul A. Patience <paul@apatience.com>
 ;;; Copyright © 2021 Ivan Gankevich <i.gankevich@spbu.ru>
 ;;; Copyright © 2021 Jean-Baptiste Volatier <jbv@pm.me>
 ;;; Copyright © 2021 Guillaume Le Vaillant <glv@posteo.net>
@@ -2008,6 +2008,9 @@ (define-public nomad-optimizer
          (add-after 'fix-sources-for-build 'fix-sources-for-tests
            (lambda _
              (substitute* "examples/CMakeLists.txt"
+               ;; This test passes only sometimes.
+               (("^ +add_subdirectory\\(\\$\\{CMAKE_CURRENT_SOURCE_DIR\\}/advanced/library/PSDMads\\)\n")
+                "")
                ;; examples/basic/batch/example3 is accidentally omitted.
                (("^(add_subdirectory\\(\\$\\{CMAKE_CURRENT_SOURCE_DIR\\}/basic/batch/example)2(\\)\n)"
                  _ prefix suffix)
@@ -2052,10 +2055,6 @@ (define-public nomad-optimizer
               "examples/advanced/library/exampleSuggestAndObserve/cache0.txt")

              (let ((builddir (string-append (getcwd) "/../build")))
-               ;; For some reason, omitting this fix causes the
-               ;; examples/advanced/library/PSDMads test to fail, even though
-               ;; it doesn't seem to reference any part of the FixedVariable
-               ;; test.
                (let ((dir "examples/advanced/library/FixedVariable"))
                  (substitute* (string-append dir "/fixedVariable.cpp")
                    (("^( +std::string sExe = ).*" _ prefix)
--
2.35.1

[bug#55067] [PATCH] gnu: nomad-optimizer: Disable failing test.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 573 bytes --]

Paul A. Patience schreef op vr 22-04-2022 om 16:20 [+0000]:
> ---
> It turns out the PSDMads test fails randomly.

Sounds like a bug, preferably it would 100% reliably report the
underlying problem, whatever it is.

> I've disabled it.

I don't think this is a good reason to disable the test.
Disabling tests whenever they fail defeats the point of tests.

Cf. with, say, clinical trials for testing safety of new medical
treatments.  If some drug fails one of the tests (sometimes or always),
then one cannot simply ignore the test.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#55067] [PATCH] gnu: nomad-optimizer: Disable failing test.

[Paul A. Patience]

On 2022-04-22 13:25:20-04:00, Maxime Devos wrote:
>> It turns out the PSDMads test fails randomly.
>
> Sounds like a bug, preferably it would 100% reliably report the
> underlying problem, whatever it is.

I agree with you, and I have already reported the issue to the
developers (along with some other issues that I addressed in the package
definition).

>> I've disabled it.
>
> I don't think this is a good reason to disable the test.
> Disabling tests whenever they fail defeats the point of tests.

This test failure wasn't getting detected in the past due to a missing
"set -o pipefail" in the test runner, which could explain why it was
missed by the developers.
I don't know enough to fix the bug myself, but I have a suspicion it is
a bug in the test rather than in the library.

Best regards,
Paul

[bug#55067] [PATCH] gnu: nomad-optimizer: Disable failing test.

[Maxime Devos]

[-- Attachment #1: Type: text/plain, Size: 696 bytes --]

Paul A. Patience schreef op vr 22-04-2022 om 19:02 [+0000]:
> On 2022-04-22 13:25:20-04:00, Maxime Devos wrote:
> > > It turns out the PSDMads test fails randomly.
> > 
> > Sounds like a bug, preferably it would 100% reliably report the
> > underlying problem, whatever it is.
> 
> I agree with you, and I have already reported the issue to the
> developers (along with some other issues that I addressed in the package
> definition).

Then I recommend adding a link to the upstream report (in a comment),
such that at the next update we know where to check if it has been
resolved, and to avoid giving the impression that just skipping tests
is acceptable.

Greetings,
Maxime.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

[bug#55067] [PATCH v3] gnu: nomad-optimizer: Disable failing test.

[Paul A. Patience]

* gnu/packages/maths.scm (nomad-optimizer)[arguments]: Disable the
examples/advanced/library/PSDMads test.
---
I've added a link to the issue I opened upstream regarding this failing test,
as suggested by Maxime Devos.

 gnu/packages/maths.scm | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index f4778179d6..448cdb4d73 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -45,7 +45,7 @@
 ;;; Copyright © 2021 Gerd Heber <gerd.heber@gmail.com>
 ;;; Copyright © 2021 Franck Pérignon <franck.perignon@univ-grenoble-alpes.fr>
 ;;; Copyright © 2021 Philip McGrath <philip@philipmcgrath.com>
-;;; Copyright © 2021 Paul A. Patience <paul@apatience.com>
+;;; Copyright © 2021-2022 Paul A. Patience <paul@apatience.com>
 ;;; Copyright © 2021 Ivan Gankevich <i.gankevich@spbu.ru>
 ;;; Copyright © 2021 Jean-Baptiste Volatier <jbv@pm.me>
 ;;; Copyright © 2021 Guillaume Le Vaillant <glv@posteo.net>
@@ -2008,6 +2008,10 @@ (define-public nomad-optimizer
          (add-after 'fix-sources-for-build 'fix-sources-for-tests
            (lambda _
              (substitute* "examples/CMakeLists.txt"
+               ;; This test passes only sometimes.
+               ;; See https://github.com/bbopt/nomad/issues/72.
+               (("^ +add_subdirectory\\(\\$\\{CMAKE_CURRENT_SOURCE_DIR\\}/advanced/library/PSDMads\\)\n")
+                "")
                ;; examples/basic/batch/example3 is accidentally omitted.
                (("^(add_subdirectory\\(\\$\\{CMAKE_CURRENT_SOURCE_DIR\\}/basic/batch/example)2(\\)\n)"
                  _ prefix suffix)
@@ -2052,10 +2056,6 @@ (define-public nomad-optimizer
               "examples/advanced/library/exampleSuggestAndObserve/cache0.txt")

              (let ((builddir (string-append (getcwd) "/../build")))
-               ;; For some reason, omitting this fix causes the
-               ;; examples/advanced/library/PSDMads test to fail, even though
-               ;; it doesn't seem to reference any part of the FixedVariable
-               ;; test.
                (let ((dir "examples/advanced/library/FixedVariable"))
                  (substitute* (string-append dir "/fixedVariable.cpp")
                    (("^( +std::string sExe = ).*" _ prefix)
--
2.35.1

bug#55067: [PATCH] gnu: nomad-optimizer: Disable failing test.

[Mathieu Othacehe]

> * gnu/packages/maths.scm (nomad-optimizer)[arguments]: Disable the
> examples/advanced/library/PSDMads test.

Pushed, thanks Paul & Maxime!

Mathieu