[bug#30801] [PATCH 0/1] Add opencv

[bug#30801] [PATCH 0/1] Add opencv

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 1181 bytes --]

This patch adds OpenCV, a computer vision library.

It combines the basic and contrib algorithms.

Some notes:

The test suite consists of an extra package, weighting 465MB
compressed. It runs very well. I think the size is worth it. It
consists of proprietary things (i.e. lena.jpg). As far as I understand,
that is OK, if it doesn't get in the final src/bin store output. Right?

CPU-optimization: I hope I have done everything right. Reading the
article from Guix HPC a second time helped a lot. So now it should be
compiled with SSE2/NEON being the minimum required instruction set, and
dispatches to other ISAs where available.

Size: Currently I load a bunch of dependencies in and have one
big  package as output. guix size is 1.1 GiB. I slightly have the
feeling someone could ask to split it in several outputs. Though having
one big output was the easiest thing first and I don't know how one
would handle inter-dependencies between the different outputs.


Björn


Björn Höfling (1):
  gnu: Add opencv.

 gnu/packages/image-processing.scm | 186 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 186 insertions(+)

-- 
2.16.2


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[bug#30801] [PATCH 1/1] gnu: Add opencv.

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 10183 bytes --]


* gnu/packages/image-processing.scm (opencv): New variable.
---
 gnu/packages/image-processing.scm | 186 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 186 insertions(+)

diff --git a/gnu/packages/image-processing.scm b/gnu/packages/image-processing.scm
index 6705a726c..c33075abc 100644
--- a/gnu/packages/image-processing.scm
+++ b/gnu/packages/image-processing.scm
@@ -5,6 +5,7 @@
 ;;; Copyright © 2014 Mark H Weaver <mhw@netris.org>
 ;;; Copyright © 2016 Eric Bavier <bavier@member.fsf.org>
 ;;; Copyright © 2018 Tobias Geerinckx-Rice <me@tobias.gr>
+;;; Copyright © 2018 Björn Höfling <bjoern.hoefling@bjoernhoefling.de>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -38,12 +39,16 @@
   #:use-module (gnu packages gnome)
   #:use-module (gnu packages graphics)
   #:use-module (gnu packages graphviz)
+  #:use-module (gnu packages gtk)
   #:use-module (gnu packages image)
   #:use-module (gnu packages maths)
   #:use-module (gnu packages perl)
+  #:use-module (gnu packages photo)
   #:use-module (gnu packages pkg-config)
+  #:use-module (gnu packages protobuf)
   #:use-module (gnu packages python)
   #:use-module (gnu packages serialization)
+  #:use-module (gnu packages video)
   #:use-module (gnu packages xiph)
   #:use-module (gnu packages xml)
   #:use-module (gnu packages xorg))
@@ -191,3 +196,184 @@ triangulation.  VTK has an extensive information visualization framework, has
 a suite of 3D interaction widgets, supports parallel processing, and
 integrates with various databases on GUI toolkits such as Qt and Tk.")
     (license license:bsd-3)))
+
+(define-public opencv
+  (package
+    (name "opencv")
+    (version "3.4.1")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append "https://github.com/opencv/opencv/archive/"
+                                  version ".zip"))
+              (file-name (string-append name "-" version ".zip"))
+              (sha256
+               (base32
+                "1g8pvnlkzzp50amd89149hqsbvsc2hq3vk1d6a9fksdcx8ra9g94"))
+              (modules '((guix build utils)))
+              (snippet
+               '(begin
+                  ;; Remove external libraries. We have all available in Guix:
+                  (delete-file-recursively "3rdparty")
+
+                  ;; Milky icon set is non-free:
+                  (delete-file-recursively "modules/highgui/src/files_Qt/Milky")
+
+                  ;; Some jars found:
+                  (for-each delete-file
+                            '("modules/java/test/pure_test/lib/junit-4.11.jar"
+                              "samples/java/sbt/sbt/sbt-launch.jar"))))))
+    (build-system cmake-build-system)
+    (arguments
+     `(#:configure-flags
+       (list "-DWITH_IPP=OFF"
+             "-DWITH_ITT=OFF"
+
+             ;; CPU-Features:
+             ;; See cmake/OpenCVCompilerOptimizations.cmake
+             ;; (CPU_ALL_OPTIMIZATIONS) for a list of all optimizations
+             ;; BASELINE is the minimum optimization all CPUs must support
+             ;;
+             ;; DISPATCH is the list of optional dispatches
+             "-DCPU_BASELINE=SSE2, NEON"
+
+             "-DCPU_DISPATCH=SSE3,SSSE3,SSE4_1,SSE4_2,AVX,AVX2"
+             "-DCPU_DISPATCH_REQUIRE=SSE3,SSSE3,SSE4_1,SSE4_2,AVX,AVX2"
+
+             "-DBUILD_PERF_TESTS=OFF"
+             "-D BUILD_TESTS=ON"
+
+             (string-append "-DOPENCV_EXTRA_MODULES_PATH="
+                            "/tmp/guix-build-opencv-" ,version ".drv-0"
+                            "/opencv-contrib/opencv_contrib-" ,version
+                            "/modules")
+
+             ;;Define test data:
+             (string-append "-DOPENCV_TEST_DATA_PATH="
+                            "/tmp/guix-build-opencv-" ,version ".drv-0"
+                            ;;"/opencv-3.4.0"
+                            "/opencv-extra/opencv_extra-" ,version
+                            "/testdata")
+
+             ;; Is ON by default and would try to rebuild 3rd-party protobuf,
+             ;; which we had removed, which would lead to an error:
+             "-DBUILD_PROTOBUF=OFF"
+
+             ;; Rebuild protobuf files, because we have a slightly different
+             ;; version than the included one. If we would not update, we
+             ;; would get a compile error later:
+             "-DPROTOBUF_UPDATE_FILES=ON"
+
+             ;; xfeatures2d disabled, because it downloads extra binaries from
+             ;; https://github.com/opencv/opencv_3rdparty
+             ;; defined in xfeatures2d/cmake/download_{vgg|bootdesc}.cmake
+             ;; Cmp this bug entry:
+             ;; https://github.com/opencv/opencv_contrib/issues/1131
+             "-DBUILD_opencv_xfeatures2d=OFF")
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'disable-broken-tests
+           (lambda _
+             ;; These tests fails with:
+             ;; vtkXOpenGLRenderWindow (0x723990): Could not find a decent config
+             ;; I think we have no OpenGL support with the Xvfb.
+             (substitute* '("modules/viz/test/test_tutorial3.cpp"
+                            "modules/viz/test/test_main.cpp"
+                            "modules/viz/test/tests_simple.cpp"
+                            "modules/viz/test/test_viz3d.cpp")
+               (("(TEST\\(Viz, )([a-z].*\\).*)" all pre post)
+                (string-append pre "DISABLED_" post)))
+
+             ;; This one fails with "unknown file: Failure"
+             ;; But I couldn't figure out which file was missing:
+             (substitute* (list (string-append
+                                 "../opencv-contrib/opencv_contrib-"
+                                 ,version
+                                 "/modules/face/test/test_face_align.cpp"))
+               (("(TEST\\(CV_Face_FacemarkKazemi, )(can_detect_landmarks\\).*)"
+                 all pre post)
+                (string-append pre "DISABLED_" post)))
+
+             ;; Failure reason: Bad accuracy
+             ;; Incorrect count of accurate poses [2nd case]: 90.000000 / 94.000000
+             (substitute* (list (string-append
+                                 "../opencv-contrib/opencv_contrib-"
+                                 ,version
+                                 "/modules/rgbd/test/test_odometry.cpp"))
+               (("(TEST\\(RGBD_Odometry_Rgbd, )(algorithmic\\).*)" all pre post)
+                (string-append pre "DISABLED_" post)))
+             #t))
+
+         ;; Idea copied from ldc.scm (ldc-bootstrap):
+         (add-after 'unpack 'unpack-submodule-sources
+           (lambda* (#:key inputs #:allow-other-keys)
+             (mkdir "../opencv-extra")
+             (mkdir "../opencv-contrib")
+             (let ((unpack (lambda (source target)
+                             (with-directory-excursion target
+                               (apply invoke "unzip"
+                                      (list (assoc-ref inputs source)))))))
+               (unpack "opencv-extra" "../opencv-extra")
+               (unpack "opencv-contrib" "../opencv-contrib"))))
+
+         (add-after 'set-paths 'add-ilmbase-include-path
+           (lambda* (#:key inputs #:allow-other-keys)
+           ;; OpenEXR propagates ilmbase, but its include files do not appear
+           ;; in the CPATH, so we need to add "$ilmbase/include/OpenEXR/" to
+           ;; the CPATH to satisfy the dependency on "ImathVec.h".
+           (setenv "CPATH"
+                   (string-append (assoc-ref inputs "ilmbase")
+                                  "/include/OpenEXR"
+                                  ":" (or (getenv "CPATH") "")))
+           #t))
+       (add-before 'check 'start-xserver
+         (lambda* (#:key inputs #:allow-other-keys)
+           (let ((xorg-server (assoc-ref inputs "xorg-server"))
+                 (disp ":1"))
+             (setenv "HOME" (getcwd))
+             (setenv "DISPLAY" disp)
+             ;; There must be a running X server and make check doesn't start one.
+             ;; Therefore we must do it.
+             (zero? (system (format #f "~a/bin/Xvfb ~a &" xorg-server disp)))))))))
+    (native-inputs
+     `(("unzip" ,unzip)
+       ("pkg-config" ,pkg-config)
+       ("xorg-server" ,xorg-server) ; For running the tests
+       ("opencv-extra"
+        ,(origin
+           (method url-fetch)
+           (uri (string-append "https://codeload.github.com/"
+                               "opencv/opencv_extra/zip/" version))
+           (file-name (string-append "opencv-extra-" version ".zip"))
+           (sha256
+            (base32 "0wfh3pvfxqydf7hsccp50npcsg37sf6fqi6cd3zkc4qil9zhpbps"))))
+       ("opencv-contrib"
+        ,(origin
+           (method url-fetch)
+           (uri (string-append "https://codeload.github.com/"
+                               "opencv/opencv_contrib/zip/" version))
+           (file-name (string-append "opencv-contrib-" version ".zip"))
+           (sha256
+           (base32 "18zm0qmjcdvg90c33gzv0ws0xdaid1xpqzz2xa9l2x12qkr6zj3p"))))))
+    (inputs `(("ffmpeg" ,ffmpeg)
+              ("libjpeg" ,libjpeg)
+              ("libpng" ,libpng)
+              ("jasper" ,jasper)
+              ("libtiff" ,libtiff)
+              ("hdf5" ,hdf5)
+              ("libgphoto2" ,libgphoto2)
+              ("libwebp" ,libwebp)
+              ("zlib" ,zlib)
+              ("gtkglext" ,gtkglext)
+              ("openexr" ,openexr)
+              ("ilmbase" ,ilmbase)
+              ("gtk+" ,gtk+-2)
+              ("python-numpy" ,python-numpy)
+              ("protobuf" ,protobuf)
+              ("vtk" ,vtk)
+              ("python" ,python)))
+    (synopsis "Computer Vision Library")
+    (description "OpenCV (Open Source Computer Vision) is a library aimed at
+real-time computer vision, including several hundred computer
+vision algorithms.")
+    (home-page "https://opencv.org/")
+    (license license:bsd-3)))
-- 
2.16.2


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[bug#30801] [PATCH 0/1] Add opencv

[Ludovic Courtès]

Hi Björn,

Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:

> The test suite consists of an extra package, weighting 465MB
> compressed. It runs very well. I think the size is worth it. It
> consists of proprietary things (i.e. lena.jpg). As far as I understand,
> that is OK, if it doesn't get in the final src/bin store output. Right?

As a rule of thumb, there should not be non-free stuff in the derivation
graph.

If there’s non-free software, that’s not OK, even if it doesn’t show up
in the output.

If it’s “just” data (pictures) that are non-free, that’s OK per the
FSDG:
<https://www.gnu.org/distros/free-system-distribution-guidelines.html#non-functional-data>.
If we could replace it with a free variant, I think we should clearly
encourage it, but lena.jpg is hardly replaceable in this context (I’d
hope it weren’t around for what it tells about CS, but that’s another
story…).

> CPU-optimization: I hope I have done everything right. Reading the
> article from Guix HPC a second time helped a lot. So now it should be
> compiled with SSE2/NEON being the minimum required instruction set, and
> dispatches to other ISAs where available.

Great.

> Size: Currently I load a bunch of dependencies in and have one
> big  package as output. guix size is 1.1 GiB. I slightly have the
> feeling someone could ask to split it in several outputs. Though having
> one big output was the easiest thing first and I don't know how one
> would handle inter-dependencies between the different outputs.

1.1G is a bit too much IMO, indeed.  :-)

How much is OpenCV itself?  If OpenCV itself is big, it would be nice to
look at what’s taking up space in there with ‘du’.  For instance, if
there are .a files, we might want to not build them and keep only shared
objects.

Splitting in separate outputs may or may not make sense.  If there are
examples or large pieces of HTML doc, introducing a “doc” and/or an
“examples” output may help.  Likewise, if there are binaries that depend
on more stuff that the library itself, introducing a “lib” output might
be a good idea.

Could you take a look?

Nitpick:

> +    (synopsis "Computer Vision Library")

No need to capitalize.

> +    (description "OpenCV (Open Source Computer Vision) is a library aimed at

I’d remove the parenthetic part.

> +real-time computer vision, including several hundred computer
> +vision algorithms.")

Bonus points if you can expound a little bit, without just listing those
algorithms though.  ;-)

Apart from that the patch looks very polished and that’s a pleasure to
review!

Thanks,
Ludo’.

[bug#30801] [PATCH 0/1] Add opencv

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 7699 bytes --]

Hi Ludo,

thanks for reviewing!

On Thu, 15 Mar 2018 22:04:54 +0100
ludo@gnu.org (Ludovic Courtès) wrote:

> Hi Björn,
> 
> Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:
> 
> > The test suite consists of an extra package, weighting 465MB
> > compressed. It runs very well. I think the size is worth it. It
> > consists of proprietary things (i.e. lena.jpg). As far as I
> > understand, that is OK, if it doesn't get in the final src/bin
> > store output. Right?  
> 
> As a rule of thumb, there should not be non-free stuff in the
> derivation graph.
> 
> If there’s non-free software, that’s not OK, even if it doesn’t show
> up in the output.
> 
> If it’s “just” data (pictures) that are non-free, that’s OK per the
> FSDG:
> <https://www.gnu.org/distros/free-system-distribution-guidelines.html#non-functional-data>.
> If we could replace it with a free variant, I think we should clearly
> encourage it, but lena.jpg is hardly replaceable in this context (I’d
> hope it weren’t around for what it tells about CS, but that’s another
> story…).

What is being downloaded is in the store of cause, but it is only
images and videos, no code. And it is not linked against.

The suite is 400 MB in size and replacing Lena with Linus or even
better pictures and videos of trees would need a fundamental idea of
the algorithms behind OpenCV, with I don't have. So for now I just leave
it as is, though I agree with your fundamental ideas :-)

 
> > CPU-optimization: I hope I have done everything right. Reading the
> > article from Guix HPC a second time helped a lot. So now it should
> > be compiled with SSE2/NEON being the minimum required instruction
> > set, and dispatches to other ISAs where available.  
> 
> Great.
> 
> > Size: Currently I load a bunch of dependencies in and have one
> > big  package as output. guix size is 1.1 GiB. I slightly have the
> > feeling someone could ask to split it in several outputs. Though
> > having one big output was the easiest thing first and I don't know
> > how one would handle inter-dependencies between the different
> > outputs.  
> 
> 1.1G is a bit too much IMO, indeed.  :-)
> 
> How much is OpenCV itself?  If OpenCV itself is big, it would be nice
> to look at what’s taking up space in there with ‘du’.  For instance,
> if there are .a files, we might want to not build them and keep only
> shared objects.
> 
> Splitting in separate outputs may or may not make sense.  If there are
> examples or large pieces of HTML doc, introducing a “doc” and/or an
> “examples” output may help.  Likewise, if there are binaries that
> depend on more stuff that the library itself, introducing a “lib”
> output might be a good idea.
> 
> Could you take a look?

I don't know which of the store items was my latest one, I'm currently
recompiling but that takes 1-2 hours to finish.

Most are 50MB, one is 265MB, I suppose that was the latest with all
dependencies and all (free) OpenCV modules. So the 1.1GB was
definitively the convex hull including all dependencies.

The whole thing is more or less only "lib".

I forgot to mention that I ignored the "doc" package: It built one when
I added doxygen as a dependency, but it did not install it. Now that I
know how to manually copy/install files, I could give that another try.
But that would add (to a :doc output, of cause), not substract.

The "bin" part is only 7 MB, so not worth mentioning. There are some
examples which I did not include, because they are not very interesting
in the compiled version, but more for understanding the programming
part from the source code.

OK, I looked into some other packages and tried to split it into pieces
(one piece for now). What I do is just add a phase after install and
copy things manually (copied from the git package definition):

+       (add-after 'install 'split
+         (lambda* (#:key outputs #:allow-other-keys)
+           (let* ((outputs-out (assoc-ref outputs "out"))
+                  (outputs-feature2d (assoc-ref outputs "feature2d"))
+                  (outputs-face  (assoc-ref outputs "face"))
+                  (outputs-core (assoc-ref outputs "core"))
+                  (libface     (string-append outputs-out "/lib/libopencv_face.so"))
+                  (libface*    (string-append outputs-face "/lib/libopencv_face.so"))
+                  (libface3.4     (string-append outputs-out "/lib/libopencv_face.so.3.4"))
+                  (libface3.4*    (string-append outputs-face "/lib/libopencv_face.so.3.4"))
+                  (libface3.4.1     (string-append outputs-out "/lib/libopencv_face.so.3.4.1"))
+                  (libface3.4.1*    (string-append outputs-face "/lib/libopencv_face.so.3.4.1")))
+
+                  
+                  (mkdir-p (string-append outputs-face "/lib"))
+                  
+                  (for-each (lambda (old new)
+                              (copy-file old new)
+                              (delete-file old)
+                              (chmod new #o555))
+                            (list libface libface3.4 libface3.4.1)
+                            (list libface* libface3.4* libface3.4.1*))
+                  )
+           #t ; TODO: Implement it.
+           ))
+       )))

The problem here is that this doesn't correct the RPATHS and I will get
problems with dependent library parts (The python modules in that case):

phase `strip' succeeded after 0.5 seconds
starting phase `validate-runpath'
validating RUNPATH of 46 binaries in "/gnu/store/f7aqk1my2bdprhgp3gxmnl9227gxf43m-opencv-3.4.1/lib"...
/gnu/store/f7aqk1my2bdprhgp3gxmnl9227gxf43m-opencv-3.4.1/lib/python3.6/site-packages/cv2.cpython-36m-x86_64-linux-gnu.so: error: depends on 'libopencv_face.so.3.4', which cannot be found in RUNPATH ("/gnu/store/f7aqk1my2bdprhgp3gxmnl9227gxf43m-opencv-3.4.1/lib" "/gnu/store/124ymrzp0dwx6qfh4r4r4763sa5k48sv-libsm-1.2.2/lib" "/gnu/store/dbdjmralkrzqn6b093hp69bjljvfr7zm-libice-1.0.9/lib" "/gnu/store/g7sak8qzk7lk06ggn38xpfv5mb8da6kk-libxt-1.1.5/lib" "/gnu/store/n6acaivs0jwiwpidjr551dhdni5kgpcr-glibc-2.26.105-g0890d5379c/lib" "/gnu/store/xfjba1kww8ngdc6nxldd8ly93nh13ayy-gcc-5.5.0-lib/lib" "/gnu/store/xfjba1kww8ngdc6nxldd8ly93nh13ayy-gcc-5.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/5.5.0/../../..")
validating RUNPATH of 7 binaries in "/gnu/store/f7aqk1my2bdprhgp3gxmnl9227gxf43m-opencv-3.4.1/bin"...
validating RUNPATH of 3 binaries in "/gnu/store/1356brvn9cjilcp2zizm9gx3vigmz74i-opencv-3.4.1-face/lib"...
phase `validate-runpath' failed after 0.3 seconds

So, somehow I have to solve dependencies between the different outputs.
I have no idea how to do that. For a :doc output, that doesn't matter,
but how can I tell Guix the dependency hierachy of different outputs?

 
> Nitpick:
> 
> > +    (synopsis "Computer Vision Library")  
> 
> No need to capitalize.

OK.

 
> > +    (description "OpenCV (Open Source Computer Vision) is a
> > library aimed at  
> 
> I’d remove the parenthetic part.

OK.

> 
> > +real-time computer vision, including several hundred computer
> > +vision algorithms.")  
> 
> Bonus points if you can expound a little bit, without just listing
> those algorithms though.  ;-)

OK, you got me. I will add more lines when the technical problems are
solved.

 
> Apart from that the patch looks very polished and that’s a pleasure to
> review!


Thanks. It took quite some rounds until that point. But good to know
you enjoyed the result :-)

Björn

PS: Happy Easter and this is not the xxxx-04-01 fool.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[bug#30801] [PATCH 0/1] Add opencv

[Ludovic Courtès]

Hello Björn,

Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:

> On Thu, 15 Mar 2018 22:04:54 +0100
> ludo@gnu.org (Ludovic Courtès) wrote:
>
>> Hi Björn,
>> 
>> Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:
>> 
>> > The test suite consists of an extra package, weighting 465MB
>> > compressed. It runs very well. I think the size is worth it. It
>> > consists of proprietary things (i.e. lena.jpg). As far as I
>> > understand, that is OK, if it doesn't get in the final src/bin
>> > store output. Right?  
>> 
>> As a rule of thumb, there should not be non-free stuff in the
>> derivation graph.
>> 
>> If there’s non-free software, that’s not OK, even if it doesn’t show
>> up in the output.
>> 
>> If it’s “just” data (pictures) that are non-free, that’s OK per the
>> FSDG:
>> <https://www.gnu.org/distros/free-system-distribution-guidelines.html#non-functional-data>.
>> If we could replace it with a free variant, I think we should clearly
>> encourage it, but lena.jpg is hardly replaceable in this context (I’d
>> hope it weren’t around for what it tells about CS, but that’s another
>> story…).
>
> What is being downloaded is in the store of cause, but it is only
> images and videos, no code. And it is not linked against.
>
> The suite is 400 MB in size and replacing Lena with Linus or even
> better pictures and videos of trees would need a fundamental idea of
> the algorithms behind OpenCV, with I don't have. So for now I just leave
> it as is, though I agree with your fundamental ideas :-)

OK, sounds good.

> Most are 50MB, one is 265MB, I suppose that was the latest with all
> dependencies and all (free) OpenCV modules. So the 1.1GB was
> definitively the convex hull including all dependencies.
>
> The whole thing is more or less only "lib".
>
> I forgot to mention that I ignored the "doc" package: It built one when
> I added doxygen as a dependency, but it did not install it. Now that I
> know how to manually copy/install files, I could give that another try.
> But that would add (to a :doc output, of cause), not substract.
>
> The "bin" part is only 7 MB, so not worth mentioning. There are some
> examples which I did not include, because they are not very interesting
> in the compiled version, but more for understanding the programming
> part from the source code.

OK.

> OK, I looked into some other packages and tried to split it into pieces
> (one piece for now). What I do is just add a phase after install and
> copy things manually (copied from the git package definition):
>
> +       (add-after 'install 'split
> +         (lambda* (#:key outputs #:allow-other-keys)
> +           (let* ((outputs-out (assoc-ref outputs "out"))
> +                  (outputs-feature2d (assoc-ref outputs "feature2d"))
> +                  (outputs-face  (assoc-ref outputs "face"))
> +                  (outputs-core (assoc-ref outputs "core"))
> +                  (libface     (string-append outputs-out "/lib/libopencv_face.so"))
> +                  (libface*    (string-append outputs-face "/lib/libopencv_face.so"))
> +                  (libface3.4     (string-append outputs-out "/lib/libopencv_face.so.3.4"))
> +                  (libface3.4*    (string-append outputs-face "/lib/libopencv_face.so.3.4"))
> +                  (libface3.4.1     (string-append outputs-out "/lib/libopencv_face.so.3.4.1"))
> +                  (libface3.4.1*    (string-append outputs-face "/lib/libopencv_face.so.3.4.1")))
> +
> +                  
> +                  (mkdir-p (string-append outputs-face "/lib"))
> +                  
> +                  (for-each (lambda (old new)
> +                              (copy-file old new)
> +                              (delete-file old)
> +                              (chmod new #o555))
> +                            (list libface libface3.4 libface3.4.1)
> +                            (list libface* libface3.4* libface3.4.1*))
> +                  )
> +           #t ; TODO: Implement it.
> +           ))
> +       )))
>
> The problem here is that this doesn't correct the RPATHS and I will get
> problems with dependent library parts (The python modules in that case):
>
> phase `strip' succeeded after 0.5 seconds
> starting phase `validate-runpath'
> validating RUNPATH of 46 binaries in "/gnu/store/f7aqk1my2bdprhgp3gxmnl9227gxf43m-opencv-3.4.1/lib"...
> /gnu/store/f7aqk1my2bdprhgp3gxmnl9227gxf43m-opencv-3.4.1/lib/python3.6/site-packages/cv2.cpython-36m-x86_64-linux-gnu.so: error: depends on 'libopencv_face.so.3.4', which cannot be found in RUNPATH ("/gnu/store/f7aqk1my2bdprhgp3gxmnl9227gxf43m-opencv-3.4.1/lib" "/gnu/store/124ymrzp0dwx6qfh4r4r4763sa5k48sv-libsm-1.2.2/lib" "/gnu/store/dbdjmralkrzqn6b093hp69bjljvfr7zm-libice-1.0.9/lib" "/gnu/store/g7sak8qzk7lk06ggn38xpfv5mb8da6kk-libxt-1.1.5/lib" "/gnu/store/n6acaivs0jwiwpidjr551dhdni5kgpcr-glibc-2.26.105-g0890d5379c/lib" "/gnu/store/xfjba1kww8ngdc6nxldd8ly93nh13ayy-gcc-5.5.0-lib/lib" "/gnu/store/xfjba1kww8ngdc6nxldd8ly93nh13ayy-gcc-5.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/5.5.0/../../..")

Right, you cannot just move .so files after the fact because
cv2.cpython-36m-x86_64-linux-gnu.so, for instance, has …/lib in its
RUNPATH, not …-lib/lib.  If you wanted to do that, the right way to do
it would be by modifying the makefiles/cmakelists so that they
explicitly install libopencv_face to …-lib/lib.

That said, it may be safer/easier to keep all libraries in the same
output.

BTW, are .a files installed?  It would probably help save space to not
install them.

Thanks for the update & have a nice week-end!

Ludo’.

[bug#30801] Add opencv

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 11111 bytes --]

Here is another update:

* Kept it in one output.
* No, there are no .a-files.
* I tried to build the documentation, but somehow it did not work.
  Postponed for the next update.
* Added more to the description.
* The new ffmpeg 4.0 caused a core dump during test phase, so I set it
  explicitly to the old 3.4.

Björn

From 0e234430fec497d3c7702e288474e39d37cb8ae8 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Bj=C3=B6rn=20H=C3=B6fling?=
 <bjoern.hoefling@bjoernhoefling.de>
Date: Fri, 16 Feb 2018 20:31:37 +0100
Subject: [PATCH] gnu: Add opencv.

* gnu/packages/image-processing.scm (opencv): New variable.
---
 gnu/packages/image-processing.scm | 200 ++++++++++++++++++++++++++++++
 1 file changed, 200 insertions(+)

diff --git a/gnu/packages/image-processing.scm b/gnu/packages/image-processing.scm
index 6705a726c..ef303e134 100644
--- a/gnu/packages/image-processing.scm
+++ b/gnu/packages/image-processing.scm
@@ -5,6 +5,7 @@
 ;;; Copyright © 2014 Mark H Weaver <mhw@netris.org>
 ;;; Copyright © 2016 Eric Bavier <bavier@member.fsf.org>
 ;;; Copyright © 2018 Tobias Geerinckx-Rice <me@tobias.gr>
+;;; Copyright © 2018 Björn Höfling <bjoern.hoefling@bjoernhoefling.de>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -38,12 +39,16 @@
   #:use-module (gnu packages gnome)
   #:use-module (gnu packages graphics)
   #:use-module (gnu packages graphviz)
+  #:use-module (gnu packages gtk)
   #:use-module (gnu packages image)
   #:use-module (gnu packages maths)
   #:use-module (gnu packages perl)
+  #:use-module (gnu packages photo)
   #:use-module (gnu packages pkg-config)
+  #:use-module (gnu packages protobuf)
   #:use-module (gnu packages python)
   #:use-module (gnu packages serialization)
+  #:use-module (gnu packages video)
   #:use-module (gnu packages xiph)
   #:use-module (gnu packages xml)
   #:use-module (gnu packages xorg))
@@ -191,3 +196,198 @@ triangulation.  VTK has an extensive information visualization framework, has
 a suite of 3D interaction widgets, supports parallel processing, and
 integrates with various databases on GUI toolkits such as Qt and Tk.")
     (license license:bsd-3)))
+
+(define-public opencv
+  (package
+    (name "opencv")
+    (version "3.4.1")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append "https://github.com/opencv/opencv/archive/"
+                                  version ".zip"))
+              (file-name (string-append name "-" version ".zip"))
+              (sha256
+               (base32
+                "1g8pvnlkzzp50amd89149hqsbvsc2hq3vk1d6a9fksdcx8ra9g94"))
+              (modules '((guix build utils)))
+              (snippet
+               '(begin
+                  ;; Remove external libraries. We have all available in Guix:
+                  (delete-file-recursively "3rdparty")
+
+                  ;; Milky icon set is non-free:
+                  (delete-file-recursively "modules/highgui/src/files_Qt/Milky")
+
+                  ;; Some jars found:
+                  (for-each delete-file
+                            '("modules/java/test/pure_test/lib/junit-4.11.jar"
+                              "samples/java/sbt/sbt/sbt-launch.jar"))))))
+    (build-system cmake-build-system)
+    (arguments
+     `(#:configure-flags
+       (list "-DWITH_IPP=OFF"
+             "-DWITH_ITT=OFF"
+
+             ;; CPU-Features:
+             ;; See cmake/OpenCVCompilerOptimizations.cmake
+             ;; (CPU_ALL_OPTIMIZATIONS) for a list of all optimizations
+             ;; BASELINE is the minimum optimization all CPUs must support
+             ;;
+             ;; DISPATCH is the list of optional dispatches
+             "-DCPU_BASELINE=SSE2, NEON"
+
+             "-DCPU_DISPATCH=SSE3,SSSE3,SSE4_1,SSE4_2,AVX,AVX2"
+             "-DCPU_DISPATCH_REQUIRE=SSE3,SSSE3,SSE4_1,SSE4_2,AVX,AVX2"
+
+             "-DBUILD_PERF_TESTS=OFF"
+             "-D BUILD_TESTS=ON"
+
+             (string-append "-DOPENCV_EXTRA_MODULES_PATH="
+                            "/tmp/guix-build-opencv-" ,version ".drv-0"
+                            "/opencv-contrib/opencv_contrib-" ,version
+                            "/modules")
+
+             ;;Define test data:
+             (string-append "-DOPENCV_TEST_DATA_PATH="
+                            "/tmp/guix-build-opencv-" ,version ".drv-0"
+                            ;;"/opencv-3.4.0"
+                            "/opencv-extra/opencv_extra-" ,version
+                            "/testdata")
+
+             ;; Is ON by default and would try to rebuild 3rd-party protobuf,
+             ;; which we had removed, which would lead to an error:
+             "-DBUILD_PROTOBUF=OFF"
+
+             ;; Rebuild protobuf files, because we have a slightly different
+             ;; version than the included one. If we would not update, we
+             ;; would get a compile error later:
+             "-DPROTOBUF_UPDATE_FILES=ON"
+
+             ;; xfeatures2d disabled, because it downloads extra binaries from
+             ;; https://github.com/opencv/opencv_3rdparty
+             ;; defined in xfeatures2d/cmake/download_{vgg|bootdesc}.cmake
+             ;; Cmp this bug entry:
+             ;; https://github.com/opencv/opencv_contrib/issues/1131
+             "-DBUILD_opencv_xfeatures2d=OFF")
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'disable-broken-tests
+           (lambda _
+             ;; These tests fails with:
+             ;; vtkXOpenGLRenderWindow (0x723990): Could not find a decent config
+             ;; I think we have no OpenGL support with the Xvfb.
+             (substitute* '("modules/viz/test/test_tutorial3.cpp"
+                            "modules/viz/test/test_main.cpp"
+                            "modules/viz/test/tests_simple.cpp"
+                            "modules/viz/test/test_viz3d.cpp")
+               (("(TEST\\(Viz, )([a-z].*\\).*)" all pre post)
+                (string-append pre "DISABLED_" post)))
+
+             ;; This one fails with "unknown file: Failure"
+             ;; But I couldn't figure out which file was missing:
+             (substitute* (list (string-append
+                                 "../opencv-contrib/opencv_contrib-"
+                                 ,version
+                                 "/modules/face/test/test_face_align.cpp"))
+               (("(TEST\\(CV_Face_FacemarkKazemi, )(can_detect_landmarks\\).*)"
+                 all pre post)
+                (string-append pre "DISABLED_" post)))
+
+             ;; Failure reason: Bad accuracy
+             ;; Incorrect count of accurate poses [2nd case]: 90.000000 / 94.000000
+             (substitute* (list (string-append
+                                 "../opencv-contrib/opencv_contrib-"
+                                 ,version
+                                 "/modules/rgbd/test/test_odometry.cpp"))
+               (("(TEST\\(RGBD_Odometry_Rgbd, )(algorithmic\\).*)" all pre post)
+                (string-append pre "DISABLED_" post)))
+             #t))
+
+         ;; Idea copied from ldc.scm (ldc-bootstrap):
+         (add-after 'unpack 'unpack-submodule-sources
+           (lambda* (#:key inputs #:allow-other-keys)
+             (mkdir "../opencv-extra")
+             (mkdir "../opencv-contrib")
+             (let ((unpack (lambda (source target)
+                             (with-directory-excursion target
+                               (apply invoke "unzip"
+                                      (list (assoc-ref inputs source)))))))
+               (unpack "opencv-extra" "../opencv-extra")
+               (unpack "opencv-contrib" "../opencv-contrib"))))
+
+         (add-after 'set-paths 'add-ilmbase-include-path
+           (lambda* (#:key inputs #:allow-other-keys)
+           ;; OpenEXR propagates ilmbase, but its include files do not appear
+           ;; in the CPATH, so we need to add "$ilmbase/include/OpenEXR/" to
+           ;; the CPATH to satisfy the dependency on "ImathVec.h".
+           (setenv "CPATH"
+                   (string-append (assoc-ref inputs "ilmbase")
+                                  "/include/OpenEXR"
+                                  ":" (or (getenv "CPATH") "")))
+           #t))
+       (add-before 'check 'start-xserver
+         (lambda* (#:key inputs #:allow-other-keys)
+           (let ((xorg-server (assoc-ref inputs "xorg-server"))
+                 (disp ":1"))
+             (setenv "HOME" (getcwd))
+             (setenv "DISPLAY" disp)
+             ;; There must be a running X server and make check doesn't start one.
+             ;; Therefore we must do it.
+             (zero? (system (format #f "~a/bin/Xvfb ~a &" xorg-server disp)))))))))
+    (native-inputs
+     `(("unzip" ,unzip)
+       ("pkg-config" ,pkg-config)
+       ("xorg-server" ,xorg-server) ; For running the tests
+       ("opencv-extra"
+        ,(origin
+           (method url-fetch)
+           (uri (string-append "https://codeload.github.com/"
+                               "opencv/opencv_extra/zip/" version))
+           (file-name (string-append "opencv-extra-" version ".zip"))
+           (sha256
+            (base32 "0wfh3pvfxqydf7hsccp50npcsg37sf6fqi6cd3zkc4qil9zhpbps"))))
+       ("opencv-contrib"
+        ,(origin
+           (method url-fetch)
+           (uri (string-append "https://codeload.github.com/"
+                               "opencv/opencv_contrib/zip/" version))
+           (file-name (string-append "opencv-contrib-" version ".zip"))
+           (sha256
+           (base32 "18zm0qmjcdvg90c33gzv0ws0xdaid1xpqzz2xa9l2x12qkr6zj3p"))))))
+    (inputs `(("libjpeg" ,libjpeg)
+              ("libpng" ,libpng)
+              ("jasper" ,jasper)
+              ;; ffmpeg 4.0 causes core dumps in tests.
+              ("ffmpeg" ,ffmpeg-3.4)
+              ("libtiff" ,libtiff)
+              ("hdf5" ,hdf5)
+              ("libgphoto2" ,libgphoto2)
+              ("libwebp" ,libwebp)
+              ("zlib" ,zlib)
+              ("gtkglext" ,gtkglext)
+              ("openexr" ,openexr)
+              ("ilmbase" ,ilmbase)
+              ("gtk+" ,gtk+-2)
+              ("python-numpy" ,python-numpy)
+              ("protobuf" ,protobuf)
+              ("vtk" ,vtk)
+              ("python" ,python)))
+    (synopsis "Computer vision library")
+    (description "OpenCV is a library aimed at
+real-time computer vision, including several hundred computer
+vision algorithms.  It can be used to do things like:
+
+@itemize
+@item image and video input and output
+@item image and video processing
+@item displaying
+@item feature recognition
+@item segmentation
+@item facial recognition
+@item stereo vision
+@item structure from motion
+@item augmented reality
+@item machine learning
+@end itemize\n")
+    (home-page "https://opencv.org/")
+    (license license:bsd-3)))
-- 
2.17.0



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

bug#30801: Add opencv

[Ludovic Courtès]

Hi Björn,

Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:

> Here is another update:
>
> * Kept it in one output.
> * No, there are no .a-files.
> * I tried to build the documentation, but somehow it did not work.
>   Postponed for the next update.
> * Added more to the description.
> * The new ffmpeg 4.0 caused a core dump during test phase, so I set it
>   explicitly to the old 3.4.

Awesome, thanks for looking into this.

> From 0e234430fec497d3c7702e288474e39d37cb8ae8 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Bj=C3=B6rn=20H=C3=B6fling?=
>  <bjoern.hoefling@bjoernhoefling.de>
> Date: Fri, 16 Feb 2018 20:31:37 +0100
> Subject: [PATCH] gnu: Add opencv.
>
> * gnu/packages/image-processing.scm (opencv): New variable.

Applied!

‘guix lint’ reports this:

  gnu/packages/image-processing.scm:201:2: opencv@3.4.1: probably vulnerable to CVE-2018-7712, CVE-2018-7713, CVE-2018-7714

Could you take a look?  It could be that 3.4.2 is around the corner and
we’ll just update at that point; if not, we may have to apply upstream
patches for these issues.

Thank you!

Ludo’.

[bug#30801] Add opencv

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 781 bytes --]

On Thu, 10 May 2018 00:01:13 +0200
ludo@gnu.org (Ludovic Courtès) wrote:


> > * gnu/packages/image-processing.scm (opencv): New variable.  
> 
> Applied!

Thanks.
 
> ‘guix lint’ reports this:
> 
>   gnu/packages/image-processing.scm:201:2: opencv@3.4.1: probably
> vulnerable to CVE-2018-7712, CVE-2018-7713, CVE-2018-7714
> 
> Could you take a look?  It could be that 3.4.2 is around the corner
> and we’ll just update at that point; if not, we may have to apply
> upstream patches for these issues.

While finally linting, I noticed these too. OpenCV claims this is not
an issue:

https://github.com/opencv/opencv/issues/10998

Should we mention it somewhere in the code? Is there a formal process
to hide or comment specific CVEs?


Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[bug#30801] Add opencv

[Ludovic Courtès]

Hello!

Björn Höfling <bjoern.hoefling@bjoernhoefling.de> skribis:

> On Thu, 10 May 2018 00:01:13 +0200
> ludo@gnu.org (Ludovic Courtès) wrote:

[...]

>> ‘guix lint’ reports this:
>> 
>>   gnu/packages/image-processing.scm:201:2: opencv@3.4.1: probably
>> vulnerable to CVE-2018-7712, CVE-2018-7713, CVE-2018-7714
>> 
>> Could you take a look?  It could be that 3.4.2 is around the corner
>> and we’ll just update at that point; if not, we may have to apply
>> upstream patches for these issues.
>
> While finally linting, I noticed these too. OpenCV claims this is not
> an issue:
>
> https://github.com/opencv/opencv/issues/10998
>
> Should we mention it somewhere in the code? Is there a formal process
> to hide or comment specific CVEs?

The developer’s reasoning makes sense to me (IOW, the CVEs should be
against the applications that don’t handle exceptions properly rather
than against OpenCV itself.)

You can use the ‘lint-hidden-cve’ property to explicitly hide them.
Please add a comment with the URL above as well.

Thanks,
Ludo’.

[bug#30801] Add opencv

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 1178 bytes --]

On Fri, 11 May 2018 14:00:05 +0200
ludo@gnu.org (Ludovic Courtès) wrote:

> >> ‘guix lint’ reports this:
> >> 
> >>   gnu/packages/image-processing.scm:201:2: opencv@3.4.1: probably
> >> vulnerable to CVE-2018-7712, CVE-2018-7713, CVE-2018-7714
> >> 
> >> Could you take a look?  It could be that 3.4.2 is around the corner
> >> and we’ll just update at that point; if not, we may have to apply
> >> upstream patches for these issues.  
> >
> > While finally linting, I noticed these too. OpenCV claims this is
> > not an issue:
> >
> > https://github.com/opencv/opencv/issues/10998
> >
> > Should we mention it somewhere in the code? Is there a formal
> > process to hide or comment specific CVEs?  
> 
> The developer’s reasoning makes sense to me (IOW, the CVEs should be
> against the applications that don’t handle exceptions properly rather
> than against OpenCV itself.)
> 
> You can use the ‘lint-hidden-cve’ property to explicitly hide them.
> Please add a comment with the URL above as well.

I added a new patch including documentation about lint-hidden-cve:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=31437

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]