[PATCH] gnu: Add dlib.

[PATCH] gnu: Add dlib.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 67 bytes --]

Hi,

This adds the dlib machine learning toolkit.

Cheers,
Marius


[-- Attachment #2: 0001-gnu-Add-dlib.patch --]
[-- Type: text/x-patch, Size: 3834 bytes --]

From 9a24f0bfd8a0375928d7accdc0ef744f4fa304a0 Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Sat, 13 Aug 2016 11:26:10 +0100
Subject: [PATCH] gnu: Add dlib.

* gnu/packages/machine-learning.scm (dlib): New variable.
---
 gnu/packages/machine-learning.scm | 55 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 54 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index f96672c..6db0333 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2015, 2016 Ricardo Wurmus <rekado@elephly.net>
 ;;; Copyright © 2016 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2016 Marius Bakke <mbakke@fastmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -32,13 +33,15 @@
   #:use-module (gnu packages compression)
   #:use-module (gnu packages dejagnu)
   #:use-module (gnu packages gcc)
+  #:use-module (gnu packages image)
   #:use-module (gnu packages maths)
   #:use-module (gnu packages perl)
   #:use-module (gnu packages pkg-config)
   #:use-module (gnu packages python)
   #:use-module (gnu packages statistics)
   #:use-module (gnu packages swig)
-  #:use-module (gnu packages xml))
+  #:use-module (gnu packages xml)
+  #:use-module (gnu packages xorg))
 
 (define-public libsvm
   (package
@@ -467,3 +470,53 @@ geometric models.")
      "This package provides functions for feed-forward neural networks with a
 single hidden layer, and for multinomial log-linear models.")
     (license (list license:gpl2+ license:gpl3+))))
+
+(define-public dlib
+  (package
+    (name "dlib")
+    (version "19.0")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append
+                    "http://dlib.net/files/dlib-" version ".tar.bz2"))
+              (sha256
+               (base32
+                "0a4xilzhb78ffpcyq2fipri80bmlgrl5ngr7r3jygdr9p7v3z4dn"))
+              (modules '((guix build utils)))
+              (snippet
+               '(begin
+                  ;; Delete ~13MB of bundled dependencies.
+                  (delete-file-recursively "dlib/external")
+                  (delete-file-recursively "docs/dlib/external")))))
+    (build-system cmake-build-system)
+    (arguments
+     `(#:phases
+       (let ((test-dir (string-append "../dlib-" ,version "/dlib/test/build")))
+         (modify-phases %standard-phases
+           (add-before 'check 'build-test-suite
+             ;; No test target, so we build and run the unit tests here.
+             (lambda _
+               (mkdir-p test-dir)
+               (chdir test-dir)
+               (zero? (system* "cmake" ".."))
+               (zero? (system* "cmake" "--build" "." "--config" "Release"))))
+           (replace 'check
+             (lambda _ (zero? (system* "./dtest" "--runall"))))
+           (add-after 'check 'ascend-to-build-directory
+             (lambda _ (chdir "../../../../build") #t))))))
+    (native-inputs
+     `(("pkg-config" ,pkg-config)))
+    (inputs
+     `(("lapack" ,lapack)
+       ("libjpeg" ,libjpeg)
+       ("libpng" ,libpng)
+       ("libx11" ,libx11)
+       ("openblas" ,openblas)
+       ("zlib" ,zlib)))
+    (synopsis "Toolkit for making machine learning and data analysis applications in C++")
+    (description
+     "Dlib is a modern C++ toolkit containing machine learning algorithms and tools.  It
+is used in both industry and academia in a wide range of domains including robotics,
+embedded devices, mobile phones, and large high performance computing environments.")
+    (home-page "http://dlib.net")
+    (license license:boost1.0)))
-- 
2.9.2

Re: [PATCH] gnu: Add dlib.

[Leo Famulari]

On Sat, Aug 13, 2016 at 06:15:59PM +0100, Marius Bakke wrote:
> * gnu/packages/machine-learning.scm (dlib): New variable.

Thanks!

> +    (version "19.0")

The released 19.1 yesterday. Will you send an updated patch using the
latest version?

> +    (arguments
> +     `(#:phases
> +       (let ((test-dir (string-append "../dlib-" ,version "/dlib/test/build")))
> +         (modify-phases %standard-phases
> +           (add-before 'check 'build-test-suite
> +             ;; No test target, so we build and run the unit tests here.
> +             (lambda _
> +               (mkdir-p test-dir)
> +               (chdir test-dir)
> +               (zero? (system* "cmake" ".."))
> +               (zero? (system* "cmake" "--build" "." "--config" "Release"))))
> +           (replace 'check
> +             (lambda _ (zero? (system* "./dtest" "--runall"))))

I think we should combine these two phases, and use
with-directory-excursion (defined in guix build utils) instead of
chdir-ing back and forth.

> +           (add-after 'check 'ascend-to-build-directory
> +             (lambda _ (chdir "../../../../build") #t))))))

Then, this phase can be removed.

What do you think?

Re: [PATCH] gnu: Add dlib.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 1187 bytes --]

Leo Famulari <leo@famulari.name> writes:

>> +    (version "19.0")
>
> The released 19.1 yesterday. Will you send an updated patch using the
> latest version?

Wow, fresh out of the oven :)

>
>> +    (arguments
>> +     `(#:phases
>> +       (let ((test-dir (string-append "../dlib-" ,version "/dlib/test/build")))
>> +         (modify-phases %standard-phases
>> +           (add-before 'check 'build-test-suite
>> +             ;; No test target, so we build and run the unit tests here.
>> +             (lambda _
>> +               (mkdir-p test-dir)
>> +               (chdir test-dir)
>> +               (zero? (system* "cmake" ".."))
>> +               (zero? (system* "cmake" "--build" "." "--config" "Release"))))
>> +           (replace 'check
>> +             (lambda _ (zero? (system* "./dtest" "--runall"))))
>
> I think we should combine these two phases, and use
> with-directory-excursion (defined in guix build utils) instead of
> chdir-ing back and forth.
>
>> +           (add-after 'check 'ascend-to-build-directory
>> +             (lambda _ (chdir "../../../../build") #t))))))
>
> Then, this phase can be removed.

That worked great, thanks!

New patch attached.


[-- Attachment #2: 0001-gnu-Add-dlib.patch --]
[-- Type: text/x-patch, Size: 3683 bytes --]

From 5e30eff1cf24b236a78cc5abed870992e84f443f Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Sat, 13 Aug 2016 11:26:10 +0100
Subject: [PATCH] gnu: Add dlib.

* gnu/packages/machine-learning.scm (dlib): New variable.
---
 gnu/packages/machine-learning.scm | 52 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 51 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index f96672c..bd52e8d 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2015, 2016 Ricardo Wurmus <rekado@elephly.net>
 ;;; Copyright © 2016 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2016 Marius Bakke <mbakke@fastmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -32,13 +33,15 @@
   #:use-module (gnu packages compression)
   #:use-module (gnu packages dejagnu)
   #:use-module (gnu packages gcc)
+  #:use-module (gnu packages image)
   #:use-module (gnu packages maths)
   #:use-module (gnu packages perl)
   #:use-module (gnu packages pkg-config)
   #:use-module (gnu packages python)
   #:use-module (gnu packages statistics)
   #:use-module (gnu packages swig)
-  #:use-module (gnu packages xml))
+  #:use-module (gnu packages xml)
+  #:use-module (gnu packages xorg))
 
 (define-public libsvm
   (package
@@ -467,3 +470,50 @@ geometric models.")
      "This package provides functions for feed-forward neural networks with a
 single hidden layer, and for multinomial log-linear models.")
     (license (list license:gpl2+ license:gpl3+))))
+
+(define-public dlib
+  (package
+    (name "dlib")
+    (version "19.1")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append
+                    "http://dlib.net/files/dlib-" version ".tar.bz2"))
+              (sha256
+               (base32
+                "0p2pvcdalc6jhb6r99ybvjd9x74sclr0ngswdg9j2xl5pj7knbr4"))
+              (modules '((guix build utils)))
+              (snippet
+               '(begin
+                  ;; Delete ~13MB of bundled dependencies.
+                  (delete-file-recursively "dlib/external")
+                  (delete-file-recursively "docs/dlib/external")))))
+    (build-system cmake-build-system)
+    (arguments
+     `(#:phases
+       (let ((test-dir (string-append "../dlib-" ,version "/dlib/test/build")))
+         (modify-phases %standard-phases
+           (replace 'check
+             ;; No test target, so we build and run the unit tests here.
+             (lambda _
+               (mkdir-p test-dir)
+               (with-directory-excursion test-dir
+                 (zero? (system* "cmake" ".."))
+                 (zero? (system* "cmake" "--build" "." "--config" "Release"))
+                 (zero? (system* "./dtest" "--runall")))))))))
+    (native-inputs
+     `(("pkg-config" ,pkg-config)))
+    (inputs
+     `(("lapack" ,lapack)
+       ("libjpeg" ,libjpeg)
+       ("libpng" ,libpng)
+       ("libx11" ,libx11)
+       ("openblas" ,openblas)
+       ("zlib" ,zlib)))
+    (synopsis "Toolkit for making machine learning and data analysis applications in C++")
+    (description
+     "Dlib is a modern C++ toolkit containing machine learning algorithms and tools.  It
+is used in both industry and academia in a wide range of domains including robotics,
+embedded devices, mobile phones, and large high performance computing environments.")
+    (home-page "http://dlib.net")
+    (license license:boost1.0)))
-- 
2.9.2

Re: [PATCH] gnu: Add dlib.

[Alex Kost]

Marius Bakke (2016-08-14 22:52 +0300) wrote:

Hello and welcome!

> From 5e30eff1cf24b236a78cc5abed870992e84f443f Mon Sep 17 00:00:00 2001
> From: Marius Bakke <mbakke@fastmail.com>
> Date: Sat, 13 Aug 2016 11:26:10 +0100
> Subject: [PATCH] gnu: Add dlib.
[...]
> +(define-public dlib
> +  (package
> +    (name "dlib")
> +    (version "19.1")
> +    (source (origin
> +              (method url-fetch)
> +              (uri (string-append
> +                    "http://dlib.net/files/dlib-" version ".tar.bz2"))
> +              (sha256
> +               (base32
> +                "0p2pvcdalc6jhb6r99ybvjd9x74sclr0ngswdg9j2xl5pj7knbr4"))
> +              (modules '((guix build utils)))
> +              (snippet
> +               '(begin
> +                  ;; Delete ~13MB of bundled dependencies.
> +                  (delete-file-recursively "dlib/external")
> +                  (delete-file-recursively "docs/dlib/external")))))
> +    (build-system cmake-build-system)
> +    (arguments
> +     `(#:phases
> +       (let ((test-dir (string-append "../dlib-" ,version "/dlib/test/build")))

I think it's better to move this 'let' inside the phase: ...

> +         (modify-phases %standard-phases
> +           (replace 'check
> +             ;; No test target, so we build and run the unit tests here.
> +             (lambda _

... here.
> +               (mkdir-p test-dir)
> +               (with-directory-excursion test-dir
> +                 (zero? (system* "cmake" ".."))
> +                 (zero? (system* "cmake" "--build" "." "--config" "Release"))
> +                 (zero? (system* "./dtest" "--runall")))))))))

There is no point in the first 2 'zero?' calls as their returning
values will be lost, rather:

  (and (zero? (system* "cmake" ".."))
       (zero? (system* "cmake" "--build" "." "--config" "Release"))
       (zero? (system* "./dtest" "--runall")))

> +    (native-inputs
> +     `(("pkg-config" ,pkg-config)))
> +    (inputs
> +     `(("lapack" ,lapack)
> +       ("libjpeg" ,libjpeg)
> +       ("libpng" ,libpng)
> +       ("libx11" ,libx11)
> +       ("openblas" ,openblas)
> +       ("zlib" ,zlib)))
> +    (synopsis "Toolkit for making machine learning and data analysis applications in C++")
> +    (description
> +     "Dlib is a modern C++ toolkit containing machine learning algorithms and tools.  It
> +is used in both industry and academia in a wide range of domains including robotics,
> +embedded devices, mobile phones, and large high performance computing environments.")

As for me, the above lines (in synopsis and description) are too long, I
would stay inside 72-78 columns.

> +    (home-page "http://dlib.net")
> +    (license license:boost1.0)))

Otherwise, the patch looks good to me, so if Leo or other people don't
have comments, I think it can be committed.

-- 
Alex

Re: [PATCH] gnu: Add dlib.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 2047 bytes --]


>> +    (build-system cmake-build-system)
>> +    (arguments
>> +     `(#:phases
>> +       (let ((test-dir (string-append "../dlib-" ,version "/dlib/test/build")))
>
> I think it's better to move this 'let' inside the phase: ...
>
>> +         (modify-phases %standard-phases
>> +           (replace 'check
>> +             ;; No test target, so we build and run the unit tests here.
>> +             (lambda _
>
> ... here.

You're right. I tried that in an early iteration and it failed; I
assumed "version" was not available in that scope. Must have noobed
something else since it works fine now.. :)

>> +               (mkdir-p test-dir)
>> +               (with-directory-excursion test-dir
>> +                 (zero? (system* "cmake" ".."))
>> +                 (zero? (system* "cmake" "--build" "." "--config" "Release"))
>> +                 (zero? (system* "./dtest" "--runall")))))))))
>
> There is no point in the first 2 'zero?' calls as their returning
> values will be lost, rather:
>
>   (and (zero? (system* "cmake" ".."))
>        (zero? (system* "cmake" "--build" "." "--config" "Release"))
>        (zero? (system* "./dtest" "--runall")))

Thanks for pointing that out; fixed.

>> +    (native-inputs
>> +     `(("pkg-config" ,pkg-config)))
>> +    (inputs
>> +     `(("lapack" ,lapack)
>> +       ("libjpeg" ,libjpeg)
>> +       ("libpng" ,libpng)
>> +       ("libx11" ,libx11)
>> +       ("openblas" ,openblas)
>> +       ("zlib" ,zlib)))
>> +    (synopsis "Toolkit for making machine learning and data analysis applications in C++")
>> +    (description
>> +     "Dlib is a modern C++ toolkit containing machine learning algorithms and tools.  It
>> +is used in both industry and academia in a wide range of domains including robotics,
>> +embedded devices, mobile phones, and large high performance computing environments.")
>
> As for me, the above lines (in synopsis and description) are too long, I
> would stay inside 72-78 columns.

Disagreeing with the linter, are we? ;) I've trimmed the lengths a bit.

Thanks!


[-- Attachment #2: 0001-gnu-Add-dlib.patch --]
[-- Type: text/x-patch, Size: 3708 bytes --]

From 8ffcd39f9c8e3cea7ae925341cc19dc3b45941a3 Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Sat, 13 Aug 2016 11:26:10 +0100
Subject: [PATCH] gnu: Add dlib.

* gnu/packages/machine-learning.scm (dlib): New variable.
---
 gnu/packages/machine-learning.scm | 54 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 53 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index f96672c..674af10 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2015, 2016 Ricardo Wurmus <rekado@elephly.net>
 ;;; Copyright © 2016 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2016 Marius Bakke <mbakke@fastmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -32,13 +33,15 @@
   #:use-module (gnu packages compression)
   #:use-module (gnu packages dejagnu)
   #:use-module (gnu packages gcc)
+  #:use-module (gnu packages image)
   #:use-module (gnu packages maths)
   #:use-module (gnu packages perl)
   #:use-module (gnu packages pkg-config)
   #:use-module (gnu packages python)
   #:use-module (gnu packages statistics)
   #:use-module (gnu packages swig)
-  #:use-module (gnu packages xml))
+  #:use-module (gnu packages xml)
+  #:use-module (gnu packages xorg))
 
 (define-public libsvm
   (package
@@ -467,3 +470,52 @@ geometric models.")
      "This package provides functions for feed-forward neural networks with a
 single hidden layer, and for multinomial log-linear models.")
     (license (list license:gpl2+ license:gpl3+))))
+
+(define-public dlib
+  (package
+    (name "dlib")
+    (version "19.1")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append
+                    "http://dlib.net/files/dlib-" version ".tar.bz2"))
+              (sha256
+               (base32
+                "0p2pvcdalc6jhb6r99ybvjd9x74sclr0ngswdg9j2xl5pj7knbr4"))
+              (modules '((guix build utils)))
+              (snippet
+               '(begin
+                  ;; Delete ~13MB of bundled dependencies.
+                  (delete-file-recursively "dlib/external")
+                  (delete-file-recursively "docs/dlib/external")))))
+    (build-system cmake-build-system)
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (replace 'check
+           (lambda _
+             ;; No test target, so we build and run the unit tests here.
+             (let ((test-dir (string-append "../dlib-" ,version "/dlib/test/build")))
+               (mkdir-p test-dir)
+               (with-directory-excursion test-dir
+                 (and (zero? (system* "cmake" ".."))
+                      (zero? (system* "cmake" "--build" "." "--config" "Release"))
+                      (zero? (system* "./dtest" "--runall"))))))))))
+    (native-inputs
+     `(("pkg-config" ,pkg-config)))
+    (inputs
+     `(("lapack" ,lapack)
+       ("libjpeg" ,libjpeg)
+       ("libpng" ,libpng)
+       ("libx11" ,libx11)
+       ("openblas" ,openblas)
+       ("zlib" ,zlib)))
+    (synopsis
+     "Toolkit for making machine learning and data analysis applications in C++")
+    (description
+     "Dlib is a modern C++ toolkit containing machine learning algorithms and
+tools.  It is used in both industry and academia in a wide range of domains
+including robotics, embedded devices, mobile phones, and large high performance
+computing environments.")
+    (home-page "http://dlib.net")
+    (license license:boost1.0)))
-- 
2.9.2

Re: [PATCH] gnu: Add dlib.

[Leo Famulari]

On Mon, Aug 15, 2016 at 12:51:15PM +0100, Marius Bakke wrote:
> Date: Sat, 13 Aug 2016 11:26:10 +0100
> Subject: [PATCH] gnu: Add dlib.
> 
> * gnu/packages/machine-learning.scm (dlib): New variable.

Thanks for the updated patch.

Does it build for you? On my x86_64 machine, it fails consistently. This
is all it prints:

Running test_empirical_kernel_map / phase `check' failed after 2043.7 seconds 

Re: [PATCH] gnu: Add dlib.

[Marius Bakke]

Leo Famulari <leo@famulari.name> writes:

> On Mon, Aug 15, 2016 at 12:51:15PM +0100, Marius Bakke wrote:
>> Date: Sat, 13 Aug 2016 11:26:10 +0100
>> Subject: [PATCH] gnu: Add dlib.
>> 
>> * gnu/packages/machine-learning.scm (dlib): New variable.
>
> Thanks for the updated patch.
>
> Does it build for you? On my x86_64 machine, it fails consistently. This
> is all it prints:
>
> Running test_empirical_kernel_map / phase `check' failed after 2043.7 seconds 

Strange, I've built this dozens of times now with no test failures
(GuixSD on amd64). Are you memory constrained by any chance? Any OOM or
segfaults? I don't think the tests have taken more than ~15 minutes to
run either, once they are built.

Will try it out on some other machines.

Re: [PATCH] gnu: Add dlib.

[Leo Famulari]

On Mon, Aug 15, 2016 at 09:29:11PM +0100, Marius Bakke wrote:
> Leo Famulari <leo@famulari.name> writes:
> > Running test_empirical_kernel_map / phase `check' failed after 2043.7 seconds 
> 
> Strange, I've built this dozens of times now with no test failures
> (GuixSD on amd64). Are you memory constrained by any chance? Any OOM or
> segfaults? I don't think the tests have taken more than ~15 minutes to
> run either, once they are built.

I don't think I'm particularly resource constrained. I have ~7 GiB RAM
available to build with, plenty of disk space, and the processor is this
thing:

http://ark.intel.com/products/64896/Intel-Core-i5-3320M-Processor-3M-Cache-up-to-3_30-GHz

It appears to exit cleanly (no segfault or OOM).

It fails the same test on this machine as well as a slower x86_64
(Thinkpad x200s). One machine is Guix / Debian, the other is GuixSD.

Re: [PATCH] gnu: Add dlib.

[Ben Woodcroft]

On 16/08/16 08:28, Leo Famulari wrote:
> On Mon, Aug 15, 2016 at 09:29:11PM +0100, Marius Bakke wrote:
>> Leo Famulari <leo@famulari.name> writes:
>>> Running test_empirical_kernel_map / phase `check' failed after 2043.7 seconds
>> Strange, I've built this dozens of times now with no test failures
>> (GuixSD on amd64). Are you memory constrained by any chance? Any OOM or
>> segfaults? I don't think the tests have taken more than ~15 minutes to
>> run either, once they are built.
> I don't think I'm particularly resource constrained. I have ~7 GiB RAM
> available to build with, plenty of disk space, and the processor is this
> thing:
>
> http://ark.intel.com/products/64896/Intel-Core-i5-3320M-Processor-3M-Cache-up-to-3_30-GHz
>
> It appears to exit cleanly (no segfault or OOM).
>
> It fails the same test on this machine as well as a slower x86_64
> (Thinkpad x200s). One machine is Guix / Debian, the other is GuixSD.
I built this patch on a x86_64 server with 40 cores and 256G of memory 
twice, here's what I got:

Running test_one_vs_all_trainer \

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!! TEST FAILED: test_one_vs_all_trainer 
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Failure message from test:

Error occurred at line 141.
Error occurred in file 
/tmp/nix-build-dlib-19.1.drv-0/dlib-19.1/dlib/test/one_vs_all_trainer.cpp.
Failing expression was ans == res.
res:
60  0  0
  0 69  1
  0  0 80




Testing Finished
Total number of individual testing statements executed: 471078543
Number of failed tests: 1
Number of passed tests: 133

phase `check' failed after 2206.2 seconds
builder for `/gnu/store/gqbzlp1g5h038rxswaz2b6y78d57q8sx-dlib-19.1.drv' 
failed with exit code 1
@ build-failed /gnu/store/gqbzlp1g5h038rxswaz2b6y78d57q8sx-dlib-19.1.drv 
- 1 builder for 
`/gnu/store/gqbzlp1g5h038rxswaz2b6y78d57q8sx-dlib-19.1.drv' failed with 
exit code 1
guix build: error: build failed: build of 
`/gnu/store/gqbzlp1g5h038rxswaz2b6y78d57q8sx-dlib-19.1.drv' failed


and in the second run.....


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!! TEST FAILED: test_one_vs_all_trainer 
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Failure message from test:

Error occurred at line 141.
Error occurred in file 
/tmp/nix-build-dlib-19.1.drv-0/dlib-19.1/dlib/test/one_vs_all_trainer.cpp.
Failing expression was ans == res.
res:
30  0 30
  0 35 35
  0  0 80




Testing Finished
Total number of individual testing statements executed: 470341624
Number of failed tests: 1
Number of passed tests: 133

phase `check' failed after 2183.7 seconds
builder for `/gnu/store/gqbzlp1g5h038rxswaz2b6y78d57q8sx-dlib-19.1.drv' 
failed with exit code 1
@ build-failed /gnu/store/gqbzlp1g5h038rxswaz2b6y78d57q8sx-dlib-19.1.drv 
- 1 builder for 
`/gnu/store/gqbzlp1g5h038rxswaz2b6y78d57q8sx-dlib-19.1.drv' failed with 
exit code 1
guix build: error: build failed: build of 
`/gnu/store/gqbzlp1g5h038rxswaz2b6y78d57q8sx-dlib-19.1.drv' failed

real    37m12.419s
user    0m5.344s
sys    0m0.412s

Re: [PATCH] gnu: Add dlib.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 2568 bytes --]

Ben Woodcroft <b.woodcroft@uq.edu.au> writes:

> On 16/08/16 08:28, Leo Famulari wrote:
>> On Mon, Aug 15, 2016 at 09:29:11PM +0100, Marius Bakke wrote:
>>> Leo Famulari <leo@famulari.name> writes:
>>>> Running test_empirical_kernel_map / phase `check' failed after 2043.7 seconds
>>> Strange, I've built this dozens of times now with no test failures
>>> (GuixSD on amd64). Are you memory constrained by any chance? Any OOM or
>>> segfaults? I don't think the tests have taken more than ~15 minutes to
>>> run either, once they are built.
>> I don't think I'm particularly resource constrained. I have ~7 GiB RAM
>> available to build with, plenty of disk space, and the processor is this
>> thing:
>>
>> http://ark.intel.com/products/64896/Intel-Core-i5-3320M-Processor-3M-Cache-up-to-3_30-GHz
>>
>> It appears to exit cleanly (no segfault or OOM).
>>
>> It fails the same test on this machine as well as a slower x86_64
>> (Thinkpad x200s). One machine is Guix / Debian, the other is GuixSD.
> I built this patch on a x86_64 server with 40 cores and 256G of memory 
> twice, here's what I got:
>
> Running test_one_vs_all_trainer \
>
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!! TEST FAILED: test_one_vs_all_trainer 
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>
> Failure message from test:
>
> Error occurred at line 141.
> Error occurred in file 
> /tmp/nix-build-dlib-19.1.drv-0/dlib-19.1/dlib/test/one_vs_all_trainer.cpp.
> Failing expression was ans == res.
> res:
> 60  0  0
>   0 69  1
>   0  0 80
>
>
>
>
> Testing Finished
> Total number of individual testing statements executed: 471078543
> Number of failed tests: 1
> Number of passed tests: 133

I can reproduce Leos error on a foreign distro (Ubuntu 16.04, 32GiB RAM)
and see this in dmesg:

[497259.758549] dtest[27102]: segfault at b50 ip 00007ffff68270e0 sp 00007fffffffa978 error 4 in libopenblasp-r0.2.15.so[7ffff5715000+18c6000]

That memory address is 128 TiB. Though it consistently builds fine on a
64GiB GuixSD machine as verified with `guix build --check`.

I initially made this package on a foreign distro without "lapack" in
inputs and have verified that dropping LAPACK makes the tests pass.

I also found some other optional dependencies after digging around the
source, as well as a recommendation to disable/enable asserts:

http://dlib.net/dlib/config.h.html

Ben, Leo: Can you try the following patch and see if that works for you?


[-- Attachment #2: 0001-gnu-Add-dlib.patch --]
[-- Type: text/x-patch, Size: 4408 bytes --]

From 7cc190763e1312536fe3ac9cd91ea3756cf66b80 Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Sat, 13 Aug 2016 11:26:10 +0100
Subject: [PATCH] gnu: Add dlib.

* gnu/packages/machine-learning.scm (dlib): New variable.
---
 gnu/packages/machine-learning.scm | 63 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 62 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index f96672c..ebf3307 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2015, 2016 Ricardo Wurmus <rekado@elephly.net>
 ;;; Copyright © 2016 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2016 Marius Bakke <mbakke@fastmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -27,18 +28,21 @@
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system r)
   #:use-module (gnu packages)
+  #:use-module (gnu packages algebra)
   #:use-module (gnu packages autotools)
   #:use-module (gnu packages boost)
   #:use-module (gnu packages compression)
   #:use-module (gnu packages dejagnu)
   #:use-module (gnu packages gcc)
+  #:use-module (gnu packages image)
   #:use-module (gnu packages maths)
   #:use-module (gnu packages perl)
   #:use-module (gnu packages pkg-config)
   #:use-module (gnu packages python)
   #:use-module (gnu packages statistics)
   #:use-module (gnu packages swig)
-  #:use-module (gnu packages xml))
+  #:use-module (gnu packages xml)
+  #:use-module (gnu packages xorg))
 
 (define-public libsvm
   (package
@@ -467,3 +471,60 @@ geometric models.")
      "This package provides functions for feed-forward neural networks with a
 single hidden layer, and for multinomial log-linear models.")
     (license (list license:gpl2+ license:gpl3+))))
+
+(define-public dlib
+  (package
+    (name "dlib")
+    (version "19.1")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append
+                    "http://dlib.net/files/dlib-" version ".tar.bz2"))
+              (sha256
+               (base32
+                "0p2pvcdalc6jhb6r99ybvjd9x74sclr0ngswdg9j2xl5pj7knbr4"))
+              (modules '((guix build utils)))
+              (snippet
+               '(begin
+                  ;; Delete ~13MB of bundled dependencies.
+                  (delete-file-recursively "dlib/external")
+                  (delete-file-recursively "docs/dlib/external")))))
+    (build-system cmake-build-system)
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'disable-asserts
+           (lambda _
+             ;; config.h recommends explicitly enabling or disabling asserts
+             ;; when building as a shared library. By default neither is set.
+             (substitute* "dlib/config.h"
+               (("^//#define DLIB_DISABLE_ASSERTS") "#define DLIB_DISABLE_ASSERTS"))))
+         (replace 'check
+           (lambda _
+             ;; No test target, so we build and run the unit tests here.
+             (let ((test-dir (string-append "../dlib-" ,version "/dlib/test/build")))
+               (mkdir-p test-dir)
+               (with-directory-excursion test-dir
+                 (and (zero? (system* "cmake" ".."))
+                      (zero? (system* "cmake" "--build" "." "--config" "Release"))
+                      (zero? (system* "./dtest" "--runall"))))))))))
+    (native-inputs
+     `(("pkg-config" ,pkg-config)))
+    (inputs
+     `(("fftw" ,fftw)
+       ("giflib" ,giflib)
+       ;("lapack" ,lapack) XXX lapack here causes test failures in some setups.
+       ("libjpeg" ,libjpeg)
+       ("libpng" ,libpng)
+       ("libx11" ,libx11)
+       ("openblas" ,openblas)
+       ("zlib" ,zlib)))
+    (synopsis
+     "Toolkit for making machine learning and data analysis applications in C++")
+    (description
+     "Dlib is a modern C++ toolkit containing machine learning algorithms and
+tools.  It is used in both industry and academia in a wide range of domains
+including robotics, embedded devices, mobile phones, and large high performance
+computing environments.")
+    (home-page "http://dlib.net")
+    (license license:boost1.0)))
-- 
2.9.2

Re: [PATCH] gnu: Add dlib.

[Leo Famulari]

On Tue, Aug 16, 2016 at 11:45:16AM +0100, Marius Bakke wrote:
> I initially made this package on a foreign distro without "lapack" in
> inputs and have verified that dropping LAPACK makes the tests pass.
> 
> I also found some other optional dependencies after digging around the
> source, as well as a recommendation to disable/enable asserts:
> 
> http://dlib.net/dlib/config.h.html
> 
> Ben, Leo: Can you try the following patch and see if that works for you?

Yes, this patch builds for me.

Re: [PATCH] gnu: Add dlib.

[Ben Woodcroft]

On 17/08/16 06:47, Leo Famulari wrote:
> On Tue, Aug 16, 2016 at 11:45:16AM +0100, Marius Bakke wrote:
>> I initially made this package on a foreign distro without "lapack" in
>> inputs and have verified that dropping LAPACK makes the tests pass.
>>
>> I also found some other optional dependencies after digging around the
>> source, as well as a recommendation to disable/enable asserts:
>>
>> http://dlib.net/dlib/config.h.html
>>
>> Ben, Leo: Can you try the following patch and see if that works for you?
> Yes, this patch builds for me.
Me too, although it appeared non-deterministic. I'm afraid I haven't 
time to see if this patch is suitable to push just now. Leo?

ben

Re: [PATCH] gnu: Add dlib.

[Leo Famulari]

On Wed, Aug 17, 2016 at 09:31:11AM +1000, Ben Woodcroft wrote:
> 
> 
> On 17/08/16 06:47, Leo Famulari wrote:
> > On Tue, Aug 16, 2016 at 11:45:16AM +0100, Marius Bakke wrote:
> > > I initially made this package on a foreign distro without "lapack" in
> > > inputs and have verified that dropping LAPACK makes the tests pass.
> > > 
> > > I also found some other optional dependencies after digging around the
> > > source, as well as a recommendation to disable/enable asserts:
> > > 
> > > http://dlib.net/dlib/config.h.html
> > > 
> > > Ben, Leo: Can you try the following patch and see if that works for you?
> > Yes, this patch builds for me.
> Me too, although it appeared non-deterministic. I'm afraid I haven't time to
> see if this patch is suitable to push just now. Leo?

How did it appear non-deterministic to you?

Re: [PATCH] gnu: Add dlib.

[Ben Woodcroft]

On 17/08/16 09:45, Leo Famulari wrote:
> On Wed, Aug 17, 2016 at 09:31:11AM +1000, Ben Woodcroft wrote:
>>
>> On 17/08/16 06:47, Leo Famulari wrote:
>>> On Tue, Aug 16, 2016 at 11:45:16AM +0100, Marius Bakke wrote:
>>>> I initially made this package on a foreign distro without "lapack" in
>>>> inputs and have verified that dropping LAPACK makes the tests pass.
>>>>
>>>> I also found some other optional dependencies after digging around the
>>>> source, as well as a recommendation to disable/enable asserts:
>>>>
>>>> http://dlib.net/dlib/config.h.html
>>>>
>>>> Ben, Leo: Can you try the following patch and see if that works for you?
>>> Yes, this patch builds for me.
>> Me too, although it appeared non-deterministic. I'm afraid I haven't time to
>> see if this patch is suitable to push just now. Leo?
> How did it appear non-deterministic to you?
Just based on guix build --check:

guix build: error: build failed: derivation 
`/gnu/store/sxybcxw64q1ajzq6dysal75ffgq6238i-dlib-19.1.drv' may not be 
deterministic: output 
`/gnu/store/il57dcii4pzii11zlixjjxxxw699bg5x-dlib-19.1' differs

I'm actually not sure, why does it say "may not be deterministic"? If it 
builds twice and the second version is different, doesn't that mean it 
is definitely not deterministic by counter-example, unless there has 
been some leakage into the build container?

I also tried building it with #parallel-build? #f and #parallel-tests? 
#f. It worked in the first round but failed the second.

Note also that the 'disable-asserts' phase should end with '#t'.

ben

Re: [PATCH] gnu: Add dlib.

[Ben Woodcroft]

On 17/08/16 13:24, Ben Woodcroft wrote:
>
>
> On 17/08/16 09:45, Leo Famulari wrote:
>> On Wed, Aug 17, 2016 at 09:31:11AM +1000, Ben Woodcroft wrote:
>>>
>>> On 17/08/16 06:47, Leo Famulari wrote:
>>>> On Tue, Aug 16, 2016 at 11:45:16AM +0100, Marius Bakke wrote:
>>>>> I initially made this package on a foreign distro without "lapack" in
>>>>> inputs and have verified that dropping LAPACK makes the tests pass.
>>>>>
>>>>> I also found some other optional dependencies after digging around 
>>>>> the
>>>>> source, as well as a recommendation to disable/enable asserts:
>>>>>
>>>>> http://dlib.net/dlib/config.h.html
>>>>>
>>>>> Ben, Leo: Can you try the following patch and see if that works 
>>>>> for you?
>>>> Yes, this patch builds for me.
>>> Me too, although it appeared non-deterministic. I'm afraid I haven't 
>>> time to
>>> see if this patch is suitable to push just now. Leo?
>> How did it appear non-deterministic to you?
> Just based on guix build --check:
>
> guix build: error: build failed: derivation 
> `/gnu/store/sxybcxw64q1ajzq6dysal75ffgq6238i-dlib-19.1.drv' may not be 
> deterministic: output 
> `/gnu/store/il57dcii4pzii11zlixjjxxxw699bg5x-dlib-19.1' differs
>
> I'm actually not sure, why does it say "may not be deterministic"? If 
> it builds twice and the second version is different, doesn't that mean 
> it is definitely not deterministic by counter-example, unless there 
> has been some leakage into the build container?
>
> I also tried building it with #parallel-build? #f and #parallel-tests? 
> #f. It worked in the first round but failed the second.
I wasn't clear here. This was with lapack uncommented out. Subsequently 
I tested with both these flags #f and lapack commented out, then it was 
deterministic with --rounds=2. Let me know if you want any further 
testing done.

Re: [PATCH] gnu: Add dlib.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 2050 bytes --]

Ben Woodcroft <b.woodcroft@uq.edu.au> writes:

> On 17/08/16 09:45, Leo Famulari wrote:
>> On Wed, Aug 17, 2016 at 09:31:11AM +1000, Ben Woodcroft wrote:
>>>
>>> On 17/08/16 06:47, Leo Famulari wrote:
>>>> On Tue, Aug 16, 2016 at 11:45:16AM +0100, Marius Bakke wrote:
>>>>> I initially made this package on a foreign distro without "lapack" in
>>>>> inputs and have verified that dropping LAPACK makes the tests pass.
>>>>>
>>>>> I also found some other optional dependencies after digging around the
>>>>> source, as well as a recommendation to disable/enable asserts:
>>>>>
>>>>> http://dlib.net/dlib/config.h.html
>>>>>
>>>>> Ben, Leo: Can you try the following patch and see if that works for you?
>>>> Yes, this patch builds for me.
>>> Me too, although it appeared non-deterministic. I'm afraid I haven't time to
>>> see if this patch is suitable to push just now. Leo?
>> How did it appear non-deterministic to you?
> Just based on guix build --check:
>
> guix build: error: build failed: derivation 
> `/gnu/store/sxybcxw64q1ajzq6dysal75ffgq6238i-dlib-19.1.drv' may not be 
> deterministic: output 
> `/gnu/store/il57dcii4pzii11zlixjjxxxw699bg5x-dlib-19.1' differs
>
> I'm actually not sure, why does it say "may not be deterministic"? If it 
> builds twice and the second version is different, doesn't that mean it 
> is definitely not deterministic by counter-example, unless there has 
> been some leakage into the build container?
>
> I also tried building it with #parallel-build? #f and #parallel-tests? 
> #f. It worked in the first round but failed the second.
>
> Note also that the 'disable-asserts' phase should end with '#t'.

I've attached a patch with a #t in the disable-asserts phase, and also
deleting the 6MB static library.

Since `guix build --rounds=2` passes, is there any reason to delay this
patch? I've built this on GuixSD and a foreign distro and naively
verified that they are the same (the .so and all headers have the same
checksum; some of the recorded cmake input paths are different though).

Thanks,
Marius


[-- Attachment #2: 0001-gnu-Add-dlib.patch --]
[-- Type: text/x-patch, Size: 4627 bytes --]

From 77f74972d095aeb08367e00c9d683137bd7a35f3 Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Sat, 13 Aug 2016 11:26:10 +0100
Subject: [PATCH] gnu: Add dlib.

* gnu/packages/machine-learning.scm (dlib): New variable.
---
 gnu/packages/machine-learning.scm | 67 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 66 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index f96672c..4332045 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2015, 2016 Ricardo Wurmus <rekado@elephly.net>
 ;;; Copyright © 2016 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2016 Marius Bakke <mbakke@fastmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -27,18 +28,21 @@
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system r)
   #:use-module (gnu packages)
+  #:use-module (gnu packages algebra)
   #:use-module (gnu packages autotools)
   #:use-module (gnu packages boost)
   #:use-module (gnu packages compression)
   #:use-module (gnu packages dejagnu)
   #:use-module (gnu packages gcc)
+  #:use-module (gnu packages image)
   #:use-module (gnu packages maths)
   #:use-module (gnu packages perl)
   #:use-module (gnu packages pkg-config)
   #:use-module (gnu packages python)
   #:use-module (gnu packages statistics)
   #:use-module (gnu packages swig)
-  #:use-module (gnu packages xml))
+  #:use-module (gnu packages xml)
+  #:use-module (gnu packages xorg))
 
 (define-public libsvm
   (package
@@ -467,3 +471,64 @@ geometric models.")
      "This package provides functions for feed-forward neural networks with a
 single hidden layer, and for multinomial log-linear models.")
     (license (list license:gpl2+ license:gpl3+))))
+
+(define-public dlib
+  (package
+    (name "dlib")
+    (version "19.1")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append
+                    "http://dlib.net/files/dlib-" version ".tar.bz2"))
+              (sha256
+               (base32
+                "0p2pvcdalc6jhb6r99ybvjd9x74sclr0ngswdg9j2xl5pj7knbr4"))
+              (modules '((guix build utils)))
+              (snippet
+               '(begin
+                  ;; Delete ~13MB of bundled dependencies.
+                  (delete-file-recursively "dlib/external")
+                  (delete-file-recursively "docs/dlib/external")))))
+    (build-system cmake-build-system)
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'disable-asserts
+           (lambda _
+             ;; config.h recommends explicitly enabling or disabling asserts
+             ;; when building as a shared library. By default neither is set.
+             (substitute* "dlib/config.h"
+               (("^//#define DLIB_DISABLE_ASSERTS") "#define DLIB_DISABLE_ASSERTS"))
+             #t))
+         (replace 'check
+           (lambda _
+             ;; No test target, so we build and run the unit tests here.
+             (let ((test-dir (string-append "../dlib-" ,version "/dlib/test/build")))
+               (mkdir-p test-dir)
+               (with-directory-excursion test-dir
+                 (and (zero? (system* "cmake" ".."))
+                      (zero? (system* "cmake" "--build" "." "--config" "Release"))
+                      (zero? (system* "./dtest" "--runall")))))))
+         (add-after 'install 'delete-static-library
+           (lambda* (#:key outputs #:allow-other-keys)
+             (delete-file (string-append (assoc-ref outputs "out") "/lib/libdlib.a")))))))
+    (native-inputs
+     `(("pkg-config" ,pkg-config)))
+    (inputs
+     `(("fftw" ,fftw)
+       ("giflib" ,giflib)
+       ;("lapack" ,lapack) XXX lapack here causes test failures in some setups.
+       ("libjpeg" ,libjpeg)
+       ("libpng" ,libpng)
+       ("libx11" ,libx11)
+       ("openblas" ,openblas)
+       ("zlib" ,zlib)))
+    (synopsis
+     "Toolkit for making machine learning and data analysis applications in C++")
+    (description
+     "Dlib is a modern C++ toolkit containing machine learning algorithms and
+tools.  It is used in both industry and academia in a wide range of domains
+including robotics, embedded devices, mobile phones, and large high performance
+computing environments.")
+    (home-page "http://dlib.net")
+    (license license:boost1.0)))
-- 
2.9.2

Re: [PATCH] gnu: Add dlib.

[Leo Famulari]

On Wed, Aug 17, 2016 at 01:24:46PM +1000, Ben Woodcroft wrote:
> On 17/08/16 09:45, Leo Famulari wrote:
> > How did it appear non-deterministic to you?
> Just based on guix build --check:
> 
> guix build: error: build failed: derivation
> `/gnu/store/sxybcxw64q1ajzq6dysal75ffgq6238i-dlib-19.1.drv' may not be
> deterministic: output
> `/gnu/store/il57dcii4pzii11zlixjjxxxw699bg5x-dlib-19.1' differs
> 
> I'm actually not sure, why does it say "may not be deterministic"? If it
> builds twice and the second version is different, doesn't that mean it is
> definitely not deterministic by counter-example, unless there has been some
> leakage into the build container?

Ah yes, that checks if the built output is bit-for-bit identical. I
thought that you meant there was some non-determinism in the build
system, like a race condition caused by parallel building. Which could
also be true.

Re: [PATCH] gnu: Add dlib.

[Leo Famulari]

On Wed, Aug 17, 2016 at 03:48:24PM +0100, Marius Bakke wrote:
> I've attached a patch with a #t in the disable-asserts phase, and also
> deleting the 6MB static library.
> 
> Since `guix build --rounds=2` passes, is there any reason to delay this
> patch? I've built this on GuixSD and a foreign distro and naively
> verified that they are the same (the .so and all headers have the same
> checksum; some of the recorded cmake input paths are different though).

I pushed the patch as 5f0ff6a9e. Hopefully dlib is still useful without
lapack. We should really figure out what the issue is and fix it :)

Re: [PATCH] gnu: Add dlib.

[Marius Bakke]

Leo Famulari <leo@famulari.name> writes:

> I pushed the patch as 5f0ff6a9e. Hopefully dlib is still useful without
> lapack. We should really figure out what the issue is and fix it :)

I noticed this fails to build on Hydra. What's worse is that the i686,
x86_64 and armhf targets fails at completely different things. armhf and
i686 exits cleanly after failing 2 and 5 tests respectively, while the
x86_64 target seems to get the segfault we saw with lapack in inputs.

What should we do? I'd prefer to keep the package so it can easily be
tested on various architectures, but can understand if it is reverted.
Perhaps we can disable substitutes or tests, to stop bothering Hydra?

I'll try reproducing it this weekend on various qemu configurations.

Re: [PATCH] gnu: Add dlib.

[Leo Famulari]

On Fri, Aug 19, 2016 at 11:52:37AM +0100, Marius Bakke wrote:
> Leo Famulari <leo@famulari.name> writes:
> 
> > I pushed the patch as 5f0ff6a9e. Hopefully dlib is still useful without
> > lapack. We should really figure out what the issue is and fix it :)
> 
> I noticed this fails to build on Hydra. What's worse is that the i686,
> x86_64 and armhf targets fails at completely different things. armhf and
> i686 exits cleanly after failing 2 and 5 tests respectively, while the
> x86_64 target seems to get the segfault we saw with lapack in inputs.
> 
> What should we do? I'd prefer to keep the package so it can easily be
> tested on various architectures, but can understand if it is reverted.
> Perhaps we can disable substitutes or tests, to stop bothering Hydra?
> 
> I'll try reproducing it this weekend on various qemu configurations.

Let us know about the results of your tests. Then we can decide what to
do.

Re: [PATCH] gnu: Add dlib.

[Ben Woodcroft]

On 21/08/16 16:17, Leo Famulari wrote:
> On Fri, Aug 19, 2016 at 11:52:37AM +0100, Marius Bakke wrote:
>> Leo Famulari <leo@famulari.name> writes:
>>
>>> I pushed the patch as 5f0ff6a9e. Hopefully dlib is still useful without
>>> lapack. We should really figure out what the issue is and fix it :)
>> I noticed this fails to build on Hydra. What's worse is that the i686,
>> x86_64 and armhf targets fails at completely different things. armhf and
>> i686 exits cleanly after failing 2 and 5 tests respectively, while the
>> x86_64 target seems to get the segfault we saw with lapack in inputs.
>>
>> What should we do? I'd prefer to keep the package so it can easily be
>> tested on various architectures, but can understand if it is reverted.
>> Perhaps we can disable substitutes or tests, to stop bothering Hydra?
>>
>> I'll try reproducing it this weekend on various qemu configurations.
> Let us know about the results of your tests. Then we can decide what to
> do.
Can this be fixed simply by using #parallel-build #f ? Doing so makes it 
reproducible for me so arguably we should be doing that anyway, even if 
it takes longer to build.

ben

Re: [PATCH] gnu: Add dlib.

[Marius Bakke]

Ben Woodcroft <b.woodcroft@uq.edu.au> writes:

> On 21/08/16 16:17, Leo Famulari wrote:
>> On Fri, Aug 19, 2016 at 11:52:37AM +0100, Marius Bakke wrote:
>>> Leo Famulari <leo@famulari.name> writes:
>>>
>>>> I pushed the patch as 5f0ff6a9e. Hopefully dlib is still useful without
>>>> lapack. We should really figure out what the issue is and fix it :)
>>> I noticed this fails to build on Hydra. What's worse is that the i686,
>>> x86_64 and armhf targets fails at completely different things. armhf and
>>> i686 exits cleanly after failing 2 and 5 tests respectively, while the
>>> x86_64 target seems to get the segfault we saw with lapack in inputs.
>>>
>>> What should we do? I'd prefer to keep the package so it can easily be
>>> tested on various architectures, but can understand if it is reverted.
>>> Perhaps we can disable substitutes or tests, to stop bothering Hydra?
>>>
>>> I'll try reproducing it this weekend on various qemu configurations.
>> Let us know about the results of your tests. Then we can decide what to
>> do.
> Can this be fixed simply by using #parallel-build #f ? Doing so makes it 
> reproducible for me so arguably we should be doing that anyway, even if 
> it takes longer to build.

Any chance you can diff the outputs? If there is a race condition or
similar it may indeed fix some problems, but it would be nice to notify
upstream about it.

I opened an issue on their bug tracker, and the suggestion was to
disable BLAS: https://github.com/davisking/dlib/issues/197

Without OpenBLAS dlib will use an internal BLAS implementation. I'm
fairly certain that will at least fix the crash on x86_64, which was
a segfault in libopenblasp-r0.2.15.so when we had LAPACK in inputs, but
seems to consistently trigger on Hydra regardless.

I got busy this weekend, but will try to reproduce the i686 errors this
week; and also check if the newer openblas in core-updates solves the
x86_64 segfault.

Stay tuned...

-marius

Re: [PATCH] gnu: Add dlib.

[Marius Bakke]

> Without OpenBLAS dlib will use an internal BLAS implementation. I'm
> fairly certain that will at least fix the crash on x86_64, which was
> a segfault in libopenblasp-r0.2.15.so when we had LAPACK in inputs, but
> seems to consistently trigger on Hydra regardless.
>
> I got busy this weekend, but will try to reproduce the i686 errors this
> week; and also check if the newer openblas in core-updates solves the
> x86_64 segfault.
>
> Stay tuned...

Short update: I can successfully reproduce the i686 test failures simply
by `guix build --system=i686-linux` on x86_64. Removing OpenBLAS from
inputs had no effect. Now to figure out what's going on..

Marius

Re: [PATCH] gnu: Add dlib.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 1872 bytes --]

Marius Bakke <mbakke@fastmail.com> writes:

>> Without OpenBLAS dlib will use an internal BLAS implementation. I'm
>> fairly certain that will at least fix the crash on x86_64, which was
>> a segfault in libopenblasp-r0.2.15.so when we had LAPACK in inputs, but
>> seems to consistently trigger on Hydra regardless.
>>
>> I got busy this weekend, but will try to reproduce the i686 errors this
>> week; and also check if the newer openblas in core-updates solves the
>> x86_64 segfault.
>>
>> Stay tuned...
>
> Short update: I can successfully reproduce the i686 test failures simply
> by `guix build --system=i686-linux` on x86_64. Removing OpenBLAS from
> inputs had no effect. Now to figure out what's going on..

There are a couple of things going on in this thread:

1. Segfault on x86_64. This seems to have been resolved simply by
updating OpenBLAS. At least, I'm no longer able to reproduce it even
with LAPACK in inputs. So, that should fix the Hydra x86_64 build.
Can the OpenBLAS update be cherry-picked to master?

2. i686 test failures. Updating OpenBLAS fixed 1/5 errors. The remaining
four are reproducible on 32-bit Ubuntu, so they do not seem Guix
related. Upstream has been notified.

3. ARM failures. I don't have ARM hardware to test on, but I'm guessing
it's similar to i686 (i.e. not directly Guix related).

Adding "#:parallel-build? #f" had no effect on tests, indeed the check
phase does not seem to use the previously built dlib; it builds it again
without parallel-build. I will try reproducing the non-reproducibility
on some higher end hardware, hopefully this week.

I've also found that FFTW is no longer used, apparently due to thread
safety issues. So I'd appreciate if the following patch can be added.
Apologies for not catching the missing reference earlier, I will be more
careful in the future (fftw was added in the last minute..).


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-dlib-Remove-unused-fftw-from-inputs.patch --]
[-- Type: text/x-patch, Size: 1276 bytes --]

From 714f38b31996e014ed0cc56391e379e2241ee26e Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Tue, 23 Aug 2016 21:17:14 +0100
Subject: [PATCH] gnu: dlib: Remove unused fftw from inputs.

* gnu/packages/machine-learning.scm (dlib)[inputs]: Remove fftw.
  (define-module): Don't include algebra.scm.
---
 gnu/packages/machine-learning.scm | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index 4332045..7669702 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -28,7 +28,6 @@
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system r)
   #:use-module (gnu packages)
-  #:use-module (gnu packages algebra)
   #:use-module (gnu packages autotools)
   #:use-module (gnu packages boost)
   #:use-module (gnu packages compression)
@@ -515,8 +514,7 @@ single hidden layer, and for multinomial log-linear models.")
     (native-inputs
      `(("pkg-config" ,pkg-config)))
     (inputs
-     `(("fftw" ,fftw)
-       ("giflib" ,giflib)
+     `(("giflib" ,giflib)
        ;("lapack" ,lapack) XXX lapack here causes test failures in some setups.
        ("libjpeg" ,libjpeg)
        ("libpng" ,libpng)
-- 
2.9.3

Re: [PATCH] gnu: Add dlib.

[Leo Famulari]

On Wed, Aug 24, 2016 at 11:26:28AM +0100, Marius Bakke wrote:
> There are a couple of things going on in this thread:
> 
> 1. Segfault on x86_64. This seems to have been resolved simply by
> updating OpenBLAS. At least, I'm no longer able to reproduce it even
> with LAPACK in inputs. So, that should fix the Hydra x86_64 build.
> Can the OpenBLAS update be cherry-picked to master?

I'd say it depends on whether the OpenBLAS users are building
successfully on core-updates, but unfortunately core-updates is
currently failing early in the bootstrap process [0]. Can you take a
look at `guix refresh -l dlib` and pick some important looking
applications to test with the updated OpenBLAS?

[0]
https://hydra.gnu.org/jobset/gnu/core-updates
http://lists.gnu.org/archive/html/guix-devel/2016-08/msg01390.html

> 2. i686 test failures. Updating OpenBLAS fixed 1/5 errors. The remaining
> four are reproducible on 32-bit Ubuntu, so they do not seem Guix
> related. Upstream has been notified.
> 
> 3. ARM failures. I don't have ARM hardware to test on, but I'm guessing
> it's similar to i686 (i.e. not directly Guix related).

Maybe dlib is 64-bit only? If that's the case, we can disable it on
those architectures.

> Adding "#:parallel-build? #f" had no effect on tests, indeed the check
> phase does not seem to use the previously built dlib; it builds it again
> without parallel-build. I will try reproducing the non-reproducibility
> on some higher end hardware, hopefully this week.

It's weird that it rebuilds the application again for the tests. Which
build is actually installed in the case of success? Is it worth an
upstream bug report?

> I've also found that FFTW is no longer used, apparently due to thread
> safety issues. So I'd appreciate if the following patch can be added.
> Apologies for not catching the missing reference earlier, I will be more
> careful in the future (fftw was added in the last minute..).

Can you hold this patch locally in your "dlib fixes" queue? It would
trigger a rebuild of dlib on Hydra, but I don't see the point when we
expect the build to fail near the end anyways.

Re: [PATCH] gnu: Add dlib.

[Marius Bakke]

Leo Famulari <leo@famulari.name> writes:

> On Wed, Aug 24, 2016 at 11:26:28AM +0100, Marius Bakke wrote:
>> There are a couple of things going on in this thread:
>> 
>> 1. Segfault on x86_64. This seems to have been resolved simply by
>> updating OpenBLAS. At least, I'm no longer able to reproduce it even
>> with LAPACK in inputs. So, that should fix the Hydra x86_64 build.
>> Can the OpenBLAS update be cherry-picked to master?
>
> I'd say it depends on whether the OpenBLAS users are building
> successfully on core-updates, but unfortunately core-updates is
> currently failing early in the bootstrap process [0]. Can you take a
> look at `guix refresh -l dlib` and pick some important looking
> applications to test with the updated OpenBLAS?

I'm currently building the following openblas dependents: `libreoffice
bamm python-biom-format clipper shogun armadillo julia` and will try to
test BLAS functionality in some of them.

>> 2. i686 test failures. Updating OpenBLAS fixed 1/5 errors. The remaining
>> four are reproducible on 32-bit Ubuntu, so they do not seem Guix
>> related. Upstream has been notified.
>> 
>> 3. ARM failures. I don't have ARM hardware to test on, but I'm guessing
>> it's similar to i686 (i.e. not directly Guix related).
>
> Maybe dlib is 64-bit only? If that's the case, we can disable it on
> those architectures.

According to the developer[0], these targets should be supported.

0: https://github.com/davisking/dlib/issues/197

We could disable tests (at least the failing ones) on these platforms
until this issue is resolved. The mips64el target on Hydra times out
after 3600 seconds on one of the tests, but seems fine up to that point.
Some of these tests are fairly CPU heavy, so the timeout may be too low.

>> Adding "#:parallel-build? #f" had no effect on tests, indeed the check
>> phase does not seem to use the previously built dlib; it builds it again
>> without parallel-build. I will try reproducing the non-reproducibility
>> on some higher end hardware, hopefully this week.
>
> It's weird that it rebuilds the application again for the tests. Which
> build is actually installed in the case of success? Is it worth an
> upstream bug report?

We rely on cmake-build-system for everything but the check phase, which
doesn't have a "proper" check target. I'm no cmake expert, but we can
probably prevent re-build by not including the main application in
test/CmakeLists.txt and instead copy/link in our already-built version:

https://github.com/davisking/dlib/blob/master/dlib/test/CMakeLists.txt#L15

>> I've also found that FFTW is no longer used, apparently due to thread
>> safety issues. So I'd appreciate if the following patch can be added.
>> Apologies for not catching the missing reference earlier, I will be more
>> careful in the future (fftw was added in the last minute..).
>
> Can you hold this patch locally in your "dlib fixes" queue? It would
> trigger a rebuild of dlib on Hydra, but I don't see the point when we
> expect the build to fail near the end anyways.

Yep, hanging on to it for now.

Re: [PATCH] gnu: Add dlib.

[Ben Woodcroft]

On 24 August 2016 3:08:58 PM GMT-04:00, Marius Bakke <mbakke@fastmail.com> wrote:

>
>>> Adding "#:parallel-build? #f" had no effect on tests, indeed the
>check
>>> phase does not seem to use the previously built dlib; it builds it
>again
>>> without parallel-build. I will try reproducing the
>non-reproducibility
>>> on some higher end hardware, hopefully this week.
>>
>> It's weird that it rebuilds the application again for the tests.
>Which
>> build is actually installed in the case of success? Is it worth an
>> upstream bug report?
>
>We rely on cmake-build-system for everything but the check phase, which
>doesn't have a "proper" check target. I'm no cmake expert, but we can
>probably prevent re-build by not including the main application in
>test/CmakeLists.txt and instead copy/link in our already-built version:
>
>https://github.com/davisking/dlib/blob/master/dlib/test/CMakeLists.txt#L15

The cmake build system has an #:out-of-tree? option, maybe set that to #f?

Ben

Re: [PATCH] gnu: Add dlib.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 2683 bytes --]

Marius Bakke <mbakke@fastmail.com> writes:

> Leo Famulari <leo@famulari.name> writes:
>
>> On Wed, Aug 24, 2016 at 11:26:28AM +0100, Marius Bakke wrote:
>>> There are a couple of things going on in this thread:
>>> 
>>> 1. Segfault on x86_64. This seems to have been resolved simply by
>>> updating OpenBLAS. At least, I'm no longer able to reproduce it even
>>> with LAPACK in inputs. So, that should fix the Hydra x86_64 build.
>>> Can the OpenBLAS update be cherry-picked to master?
>>
>> I'd say it depends on whether the OpenBLAS users are building
>> successfully on core-updates, but unfortunately core-updates is
>> currently failing early in the bootstrap process [0]. Can you take a
>> look at `guix refresh -l dlib` and pick some important looking
>> applications to test with the updated OpenBLAS?
>
> I'm currently building the following openblas dependents: `libreoffice
> bamm python-biom-format clipper shogun armadillo julia` and will try to
> test BLAS functionality in some of them.

Shogun failed to build in this run. I don't have time to investigate
further, so picking the OpenBLAS update is not very appealing.

Instead I opted to disable the test that fails with lapack (and without,
on Hydra), since it's one specific openblas operation that is not unique
to dlib. I think it's an acceptable tradeoff, to give users the full
dlib functionality, and have the segfault "sort itself" when
core-updates lands in master.

>>> 2. i686 test failures. Updating OpenBLAS fixed 1/5 errors. The remaining
>>> four are reproducible on 32-bit Ubuntu, so they do not seem Guix
>>> related. Upstream has been notified.
>>> 
>>> 3. ARM failures. I don't have ARM hardware to test on, but I'm guessing
>>> it's similar to i686 (i.e. not directly Guix related).
>>
>> Maybe dlib is 64-bit only? If that's the case, we can disable it on
>> those architectures.
>
> According to the developer[0], these targets should be supported.
>
> 0: https://github.com/davisking/dlib/issues/197
>
> We could disable tests (at least the failing ones) on these platforms
> until this issue is resolved. The mips64el target on Hydra times out
> after 3600 seconds on one of the tests, but seems fine up to that point.
> Some of these tests are fairly CPU heavy, so the timeout may be too low.

Below is a patch which disables these tests (and the above segfault) for
19.1, rather than backporting the patches from dlib master branch.

One note about the patch: I could not figure out how to pass the list of
tests as arguments to `substitute*`, so currently it calls `substitute*`
for each of them. Any tips to prevent this?

It also no longer builds the main application twice for tests.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-dlib-Remove-unused-fftw-from-inputs.patch --]
[-- Type: text/x-patch, Size: 1280 bytes --]

From fe3b7409d32bbfd5c263d1d8f080b160ac676b84 Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Tue, 23 Aug 2016 21:17:14 +0100
Subject: [PATCH 1/2] gnu: dlib: Remove unused fftw from inputs.

* gnu/packages/machine-learning.scm (dlib)[inputs]: Remove fftw.
  (define-module): Don't include algebra.scm.
---
 gnu/packages/machine-learning.scm | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index 4332045..7669702 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -28,7 +28,6 @@
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system r)
   #:use-module (gnu packages)
-  #:use-module (gnu packages algebra)
   #:use-module (gnu packages autotools)
   #:use-module (gnu packages boost)
   #:use-module (gnu packages compression)
@@ -515,8 +514,7 @@ single hidden layer, and for multinomial log-linear models.")
     (native-inputs
      `(("pkg-config" ,pkg-config)))
     (inputs
-     `(("fftw" ,fftw)
-       ("giflib" ,giflib)
+     `(("giflib" ,giflib)
        ;("lapack" ,lapack) XXX lapack here causes test failures in some setups.
        ("libjpeg" ,libjpeg)
        ("libpng" ,libpng)
-- 
2.9.3


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-gnu-dlib-Disable-failing-tests.patch --]
[-- Type: text/x-patch, Size: 2778 bytes --]

From 121895b2d7915b1f1ef0430921418783019393d0 Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Sat, 27 Aug 2016 17:23:58 +0100
Subject: [PATCH 2/2] gnu: dlib: Disable failing tests.

* gnu/packages/machine-learning.scm (dlib)[arguments]: Add
  'disable-failing-tests phase and prevent building dlib twice.
  [inputs]: Add lapack.
---
 gnu/packages/machine-learning.scm | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)

diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index 7669702..35f1514 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -499,14 +499,25 @@ single hidden layer, and for multinomial log-linear models.")
              (substitute* "dlib/config.h"
                (("^//#define DLIB_DISABLE_ASSERTS") "#define DLIB_DISABLE_ASSERTS"))
              #t))
+         (add-after 'disable-asserts 'disable-failing-tests
+           ;; A number of tests are known to fail on 32-bit platforms in 19.1.
+           ;; See https://github.com/davisking/dlib/issues/197 for details.
+           (lambda _
+             (for-each
+              (lambda (test)
+                (substitute* "dlib/test/makefile"
+                  (((string-append "SRC \\+= " test "\\.cpp")) "")) #t)
+              (list "learning_to_track" "max_cost_assignment" ; armhf
+                    "optimization" "matrix2" "mpc" ; i686
+                    "empirical_map" ; may segfault with < openblas-0.2.18
+                    "object_detector")))) ; timeout on mips64el
          (replace 'check
            (lambda _
              ;; No test target, so we build and run the unit tests here.
-             (let ((test-dir (string-append "../dlib-" ,version "/dlib/test/build")))
-               (mkdir-p test-dir)
+             (let ((test-dir (string-append "../dlib-" ,version "/dlib/test")))
                (with-directory-excursion test-dir
-                 (and (zero? (system* "cmake" ".."))
-                      (zero? (system* "cmake" "--build" "." "--config" "Release"))
+                 (setenv "CXXFLAGS" "-std=gnu++11")
+                 (and (zero? (system* "make" "-j" (number->string (parallel-job-count))))
                       (zero? (system* "./dtest" "--runall")))))))
          (add-after 'install 'delete-static-library
            (lambda* (#:key outputs #:allow-other-keys)
@@ -515,7 +526,7 @@ single hidden layer, and for multinomial log-linear models.")
      `(("pkg-config" ,pkg-config)))
     (inputs
      `(("giflib" ,giflib)
-       ;("lapack" ,lapack) XXX lapack here causes test failures in some setups.
+       ("lapack" ,lapack)
        ("libjpeg" ,libjpeg)
        ("libpng" ,libpng)
        ("libx11" ,libx11)
-- 
2.9.3

Re: [PATCH] gnu: Add dlib.

[Leo Famulari]

On Tue, Aug 30, 2016 at 03:43:05PM +0100, Marius Bakke wrote:
> Shogun failed to build in this run. I don't have time to investigate
> further, so picking the OpenBLAS update is not very appealing.
> 
> Instead I opted to disable the test that fails with lapack (and without,
> on Hydra), since it's one specific openblas operation that is not unique
> to dlib. I think it's an acceptable tradeoff, to give users the full
> dlib functionality, and have the segfault "sort itself" when
> core-updates lands in master.

Okay, this sounds fine to me.

> Below is a patch which disables these tests (and the above segfault) for
> 19.1, rather than backporting the patches from dlib master branch.
> 
> One note about the patch: I could not figure out how to pass the list of
> tests as arguments to `substitute*`, so currently it calls `substitute*`
> for each of them. Any tips to prevent this?

Not from me — Calling all seasoned Schemers to thread :) If nobody
replies I will say this solution is fine.

Changing the subject, you could disable the tests per-architecture. Look
for uses of current-target-system and current-system for usage examples.
But this is not absolutely required, IMO.

> It also no longer builds the main application twice for tests.

Thank you :)

Re: [PATCH] gnu: Add dlib.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 428 bytes --]

Leo Famulari <leo@famulari.name> writes:

> Changing the subject, you could disable the tests per-architecture. Look
> for uses of current-target-system and current-system for usage examples.
> But this is not absolutely required, IMO.

This latest patch series now disables tests on a per-arch basis.

There may be others on MIPS failing after the timeout, but extending the
list should be obvious.

Thanks for your patience!


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: dlib-fixes.patch --]
[-- Type: text/x-patch, Size: 5625 bytes --]

From 331d291cabcb5d17d0284414cae1f2ac5c32736a Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Tue, 23 Aug 2016 21:17:14 +0100
Subject: [PATCH 1/3] gnu: dlib: Remove unused fftw from inputs.

* gnu/packages/machine-learning.scm (dlib)[inputs]: Remove fftw.
  (define-module): Don't include algebra.scm.
---
 gnu/packages/machine-learning.scm | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index 4332045..7669702 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -28,7 +28,6 @@
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system r)
   #:use-module (gnu packages)
-  #:use-module (gnu packages algebra)
   #:use-module (gnu packages autotools)
   #:use-module (gnu packages boost)
   #:use-module (gnu packages compression)
@@ -515,8 +514,7 @@ single hidden layer, and for multinomial log-linear models.")
     (native-inputs
      `(("pkg-config" ,pkg-config)))
     (inputs
-     `(("fftw" ,fftw)
-       ("giflib" ,giflib)
+     `(("giflib" ,giflib)
        ;("lapack" ,lapack) XXX lapack here causes test failures in some setups.
        ("libjpeg" ,libjpeg)
        ("libpng" ,libpng)
-- 
2.9.3


From 5c78ac7002cd92a2bfd8efa0f9902288dffe6a89 Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Sat, 27 Aug 2016 17:23:58 +0100
Subject: [PATCH 2/3] gnu: dlib: Do not build dlib twice for tests.

* gnu/packages/machine-learning.scm (dlib)[arguments]: Use makefile
  instead of cmake in check phase to prevent full rebuild.
---
 gnu/packages/machine-learning.scm | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index 7669702..e94ffdd 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -502,11 +502,10 @@ single hidden layer, and for multinomial log-linear models.")
          (replace 'check
            (lambda _
              ;; No test target, so we build and run the unit tests here.
-             (let ((test-dir (string-append "../dlib-" ,version "/dlib/test/build")))
-               (mkdir-p test-dir)
+             (let ((test-dir (string-append "../dlib-" ,version "/dlib/test")))
                (with-directory-excursion test-dir
-                 (and (zero? (system* "cmake" ".."))
-                      (zero? (system* "cmake" "--build" "." "--config" "Release"))
+                 (setenv "CXXFLAGS" "-std=gnu++11")
+                 (and (zero? (system* "make" "-j" (number->string (parallel-job-count))))
                       (zero? (system* "./dtest" "--runall")))))))
          (add-after 'install 'delete-static-library
            (lambda* (#:key outputs #:allow-other-keys)
-- 
2.9.3


From dd8e87ecfb1f475bd0f88338f233883d02e649af Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Fri, 9 Sep 2016 01:51:23 +0100
Subject: [PATCH 3/3] gnu: dlib: Disable failing tests.

* gnu/packages/machine-learning.scm (dlib)[arguments]: Add phase to
  disable tests that are known not to work in the current release.
  [inputs]: Add lapack.
---
 gnu/packages/machine-learning.scm | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index e94ffdd..dd260c9 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -499,6 +499,29 @@ single hidden layer, and for multinomial log-linear models.")
              (substitute* "dlib/config.h"
                (("^//#define DLIB_DISABLE_ASSERTS") "#define DLIB_DISABLE_ASSERTS"))
              #t))
+         (add-after 'disable-asserts 'disable-failing-tests
+           (lambda _
+             ;; One test times out on MIPS, so we need to disable it.
+             ;; The rest is known to fail on non-x86_64 platforms in the current release.
+             ;; Some have been fixed in git; this list should be readjusted next update.
+             (let* ((system ,(or (%current-target-system)
+                                 (%current-system)))
+                    (failing-tests (cond
+                                    ((string-prefix? "mips64" system)
+                                     '("object_detector" ; timeout
+                                       "data_io"))
+                                    ((string-prefix? "armhf" system)
+                                     '("learning_to_track" "max_cost_assignment"))
+                                    ((string-prefix? "i686" system)
+                                     '("optimization" "matrix2" "mpc"))
+                                    (else '()))))
+               ;; The following test fails due a bug in openblas < 0.2.18.
+               (append! failing-tests '("empirical_map"))
+               (for-each
+                (lambda (test)
+                  (substitute* "dlib/test/makefile"
+                    (((string-append "SRC \\+= " test "\\.cpp")) "")) #t)
+                failing-tests))))
          (replace 'check
            (lambda _
              ;; No test target, so we build and run the unit tests here.
@@ -514,7 +537,7 @@ single hidden layer, and for multinomial log-linear models.")
      `(("pkg-config" ,pkg-config)))
     (inputs
      `(("giflib" ,giflib)
-       ;("lapack" ,lapack) XXX lapack here causes test failures in some setups.
+       ("lapack" ,lapack)
        ("libjpeg" ,libjpeg)
        ("libpng" ,libpng)
        ("libx11" ,libx11)
-- 
2.9.3

Re: [PATCH] gnu: Add dlib.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 603 bytes --]

Marius Bakke <mbakke@fastmail.com> writes:

> Leo Famulari <leo@famulari.name> writes:
>
>> Changing the subject, you could disable the tests per-architecture. Look
>> for uses of current-target-system and current-system for usage examples.
>> But this is not absolutely required, IMO.
>
> This latest patch series now disables tests on a per-arch basis.
>
> There may be others on MIPS failing after the timeout, but extending the
> list should be obvious.

Here are the patches again, with the "failing-tests" variable renamed to
"disabled-tests". Let me know if I can provide anything else. Thanks!


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-dlib-Remove-unused-fftw-from-inputs.patch --]
[-- Type: text/x-patch, Size: 1280 bytes --]

From 331d291cabcb5d17d0284414cae1f2ac5c32736a Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Tue, 23 Aug 2016 21:17:14 +0100
Subject: [PATCH 1/3] gnu: dlib: Remove unused fftw from inputs.

* gnu/packages/machine-learning.scm (dlib)[inputs]: Remove fftw.
  (define-module): Don't include algebra.scm.
---
 gnu/packages/machine-learning.scm | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index 4332045..7669702 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -28,7 +28,6 @@
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system r)
   #:use-module (gnu packages)
-  #:use-module (gnu packages algebra)
   #:use-module (gnu packages autotools)
   #:use-module (gnu packages boost)
   #:use-module (gnu packages compression)
@@ -515,8 +514,7 @@ single hidden layer, and for multinomial log-linear models.")
     (native-inputs
      `(("pkg-config" ,pkg-config)))
     (inputs
-     `(("fftw" ,fftw)
-       ("giflib" ,giflib)
+     `(("giflib" ,giflib)
        ;("lapack" ,lapack) XXX lapack here causes test failures in some setups.
        ("libjpeg" ,libjpeg)
        ("libpng" ,libpng)
-- 
2.9.3


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-gnu-dlib-Do-not-build-dlib-twice-for-tests.patch --]
[-- Type: text/x-patch, Size: 1568 bytes --]

From 5c78ac7002cd92a2bfd8efa0f9902288dffe6a89 Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Sat, 27 Aug 2016 17:23:58 +0100
Subject: [PATCH 2/3] gnu: dlib: Do not build dlib twice for tests.

* gnu/packages/machine-learning.scm (dlib)[arguments]: Use makefile
  instead of cmake in check phase to prevent full rebuild.
---
 gnu/packages/machine-learning.scm | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index 7669702..e94ffdd 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -502,11 +502,10 @@ single hidden layer, and for multinomial log-linear models.")
          (replace 'check
            (lambda _
              ;; No test target, so we build and run the unit tests here.
-             (let ((test-dir (string-append "../dlib-" ,version "/dlib/test/build")))
-               (mkdir-p test-dir)
+             (let ((test-dir (string-append "../dlib-" ,version "/dlib/test")))
                (with-directory-excursion test-dir
-                 (and (zero? (system* "cmake" ".."))
-                      (zero? (system* "cmake" "--build" "." "--config" "Release"))
+                 (setenv "CXXFLAGS" "-std=gnu++11")
+                 (and (zero? (system* "make" "-j" (number->string (parallel-job-count))))
                       (zero? (system* "./dtest" "--runall")))))))
          (add-after 'install 'delete-static-library
            (lambda* (#:key outputs #:allow-other-keys)
-- 
2.9.3


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-gnu-dlib-Disable-failing-tests.patch --]
[-- Type: text/x-patch, Size: 2786 bytes --]

From 268d1085beb3195f38e4d96e0bfee55e456d3093 Mon Sep 17 00:00:00 2001
From: Marius Bakke <mbakke@fastmail.com>
Date: Fri, 9 Sep 2016 01:51:23 +0100
Subject: [PATCH 3/3] gnu: dlib: Disable failing tests.

* gnu/packages/machine-learning.scm (dlib)[arguments]: Add phase to
  disable tests that are known not to work in the current release.
  [inputs]: Add lapack.
---
 gnu/packages/machine-learning.scm | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index e94ffdd..b663b4f 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -499,6 +499,29 @@ single hidden layer, and for multinomial log-linear models.")
              (substitute* "dlib/config.h"
                (("^//#define DLIB_DISABLE_ASSERTS") "#define DLIB_DISABLE_ASSERTS"))
              #t))
+         (add-after 'disable-asserts 'disable-failing-tests
+           (lambda _
+             ;; One test times out on MIPS, so we need to disable it.
+             ;; The rest is known to fail on non-x86_64 platforms in the current release.
+             ;; Some have been fixed in git; this list should be readjusted next update.
+             (let* ((system ,(or (%current-target-system)
+                                 (%current-system)))
+                    (disabled-tests (cond
+                                     ((string-prefix? "mips64" system)
+                                      '("object_detector" ; timeout
+                                        "data_io"))
+                                     ((string-prefix? "armhf" system)
+                                      '("learning_to_track" "max_cost_assignment"))
+                                     ((string-prefix? "i686" system)
+                                      '("optimization" "matrix2" "mpc"))
+                                     (else '()))))
+               ;; The following test fails due a bug in openblas < 0.2.18.
+               (append! disabled-tests '("empirical_map"))
+               (for-each
+                (lambda (test)
+                  (substitute* "dlib/test/makefile"
+                    (((string-append "SRC \\+= " test "\\.cpp")) "")) #t)
+                disabled-tests))))
          (replace 'check
            (lambda _
              ;; No test target, so we build and run the unit tests here.
@@ -514,7 +537,7 @@ single hidden layer, and for multinomial log-linear models.")
      `(("pkg-config" ,pkg-config)))
     (inputs
      `(("giflib" ,giflib)
-       ;("lapack" ,lapack) XXX lapack here causes test failures in some setups.
+       ("lapack" ,lapack)
        ("libjpeg" ,libjpeg)
        ("libpng" ,libpng)
        ("libx11" ,libx11)
-- 
2.9.3

Re: [PATCH] gnu: Add dlib.

[Leo Famulari]

On Sat, Sep 10, 2016 at 01:32:22PM +0100, Marius Bakke wrote:
> Marius Bakke <mbakke@fastmail.com> writes:
> 
> > Leo Famulari <leo@famulari.name> writes:
> >
> >> Changing the subject, you could disable the tests per-architecture. Look
> >> for uses of current-target-system and current-system for usage examples.
> >> But this is not absolutely required, IMO.
> >
> > This latest patch series now disables tests on a per-arch basis.
> >
> > There may be others on MIPS failing after the timeout, but extending the
> > list should be obvious.
> 
> Here are the patches again, with the "failing-tests" variable renamed to
> "disabled-tests". Let me know if I can provide anything else. Thanks!

Looks reasonable to me. Does anyone else have any comments?

Re: [PATCH] gnu: Add dlib.

[Leo Famulari]

On Sat, Sep 10, 2016 at 02:16:09PM -0400, Leo Famulari wrote:
> On Sat, Sep 10, 2016 at 01:32:22PM +0100, Marius Bakke wrote:
> > Marius Bakke <mbakke@fastmail.com> writes:
> > 
> > > Leo Famulari <leo@famulari.name> writes:
> > >
> > >> Changing the subject, you could disable the tests per-architecture. Look
> > >> for uses of current-target-system and current-system for usage examples.
> > >> But this is not absolutely required, IMO.
> > >
> > > This latest patch series now disables tests on a per-arch basis.
> > >
> > > There may be others on MIPS failing after the timeout, but extending the
> > > list should be obvious.
> > 
> > Here are the patches again, with the "failing-tests" variable renamed to
> > "disabled-tests". Let me know if I can provide anything else. Thanks!
> 
> Looks reasonable to me. Does anyone else have any comments?

Okay, pushed! Thanks Marius!

Re: [PATCH] gnu: Add dlib.

[Leo Famulari]

On Sat, Sep 10, 2016 at 01:32:22PM +0100, Marius Bakke wrote:
> +         (add-after 'disable-asserts 'disable-failing-tests
> +           (lambda _
> +             ;; One test times out on MIPS, so we need to disable it.
> +             ;; The rest is known to fail on non-x86_64 platforms in the current release.
> +             ;; Some have been fixed in git; this list should be readjusted next update.
> +             (let* ((system ,(or (%current-target-system)
> +                                 (%current-system)))
> +                    (disabled-tests (cond
> +                                     ((string-prefix? "mips64" system)
> +                                      '("object_detector" ; timeout
> +                                        "data_io"))
> +                                     ((string-prefix? "armhf" system)
> +                                      '("learning_to_track" "max_cost_assignment"))
> +                                     ((string-prefix? "i686" system)
> +                                      '("optimization" "matrix2" "mpc"))
> +                                     (else '()))))
> +               ;; The following test fails due a bug in openblas < 0.2.18.

I updated OpenBLAS to 0.2.19 on core-updates.