From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:40454) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1f2Oxk-00024m-Uf for guix-patches@gnu.org; Sat, 31 Mar 2018 18:27:07 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1f2Oxi-0002Wh-JL for guix-patches@gnu.org; Sat, 31 Mar 2018 18:27:04 -0400 Received: from debbugs.gnu.org ([208.118.235.43]:53695) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1f2Oxi-0002WV-F4 for guix-patches@gnu.org; Sat, 31 Mar 2018 18:27:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1f2Oxi-0005hJ-4E for guix-patches@gnu.org; Sat, 31 Mar 2018 18:27:02 -0400 Subject: [bug#30801] [PATCH 0/1] Add opencv Resent-Message-ID: Date: Sun, 1 Apr 2018 00:26:49 +0200 From: =?UTF-8?Q?Bj=C3=B6rn_?= =?UTF-8?Q?H=C3=B6fling?= Message-ID: <20180401002649.37231b47@alma-ubu> In-Reply-To: <87po45rqx5.fsf@gnu.org> References: <20180313175809.7d782c1a@alma-ubu> <87po45rqx5.fsf@gnu.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/mK8Z_wg5wMnG=5wW6+94PrG"; protocol="application/pgp-signature" List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+kyle=kyleam.com@gnu.org Sender: "Guix-patches" To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 30801@debbugs.gnu.org --Sig_/mK8Z_wg5wMnG=5wW6+94PrG Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Ludo, thanks for reviewing! On Thu, 15 Mar 2018 22:04:54 +0100 ludo@gnu.org (Ludovic Court=C3=A8s) wrote: > Hi Bj=C3=B6rn, >=20 > Bj=C3=B6rn H=C3=B6fling skribis: >=20 > > The test suite consists of an extra package, weighting 465MB > > compressed. It runs very well. I think the size is worth it. It > > consists of proprietary things (i.e. lena.jpg). As far as I > > understand, that is OK, if it doesn't get in the final src/bin > > store output. Right? =20 >=20 > As a rule of thumb, there should not be non-free stuff in the > derivation graph. >=20 > If there=E2=80=99s non-free software, that=E2=80=99s not OK, even if it d= oesn=E2=80=99t show > up in the output. >=20 > If it=E2=80=99s =E2=80=9Cjust=E2=80=9D data (pictures) that are non-free,= that=E2=80=99s OK per the > FSDG: > . > If we could replace it with a free variant, I think we should clearly > encourage it, but lena.jpg is hardly replaceable in this context (I=E2=80= =99d > hope it weren=E2=80=99t around for what it tells about CS, but that=E2=80= =99s another > story=E2=80=A6). What is being downloaded is in the store of cause, but it is only images and videos, no code. And it is not linked against. The suite is 400 MB in size and replacing Lena with Linus or even better pictures and videos of trees would need a fundamental idea of the algorithms behind OpenCV, with I don't have. So for now I just leave it as is, though I agree with your fundamental ideas :-) =20 > > CPU-optimization: I hope I have done everything right. Reading the > > article from Guix HPC a second time helped a lot. So now it should > > be compiled with SSE2/NEON being the minimum required instruction > > set, and dispatches to other ISAs where available. =20 >=20 > Great. >=20 > > Size: Currently I load a bunch of dependencies in and have one > > big package as output. guix size is 1.1 GiB. I slightly have the > > feeling someone could ask to split it in several outputs. Though > > having one big output was the easiest thing first and I don't know > > how one would handle inter-dependencies between the different > > outputs. =20 >=20 > 1.1G is a bit too much IMO, indeed. :-) >=20 > How much is OpenCV itself? If OpenCV itself is big, it would be nice > to look at what=E2=80=99s taking up space in there with =E2=80=98du=E2=80= =99. For instance, > if there are .a files, we might want to not build them and keep only > shared objects. >=20 > Splitting in separate outputs may or may not make sense. If there are > examples or large pieces of HTML doc, introducing a =E2=80=9Cdoc=E2=80=9D= and/or an > =E2=80=9Cexamples=E2=80=9D output may help. Likewise, if there are binar= ies that > depend on more stuff that the library itself, introducing a =E2=80=9Clib= =E2=80=9D > output might be a good idea. >=20 > Could you take a look? I don't know which of the store items was my latest one, I'm currently recompiling but that takes 1-2 hours to finish. Most are 50MB, one is 265MB, I suppose that was the latest with all dependencies and all (free) OpenCV modules. So the 1.1GB was definitively the convex hull including all dependencies. The whole thing is more or less only "lib". I forgot to mention that I ignored the "doc" package: It built one when I added doxygen as a dependency, but it did not install it. Now that I know how to manually copy/install files, I could give that another try. But that would add (to a :doc output, of cause), not substract. The "bin" part is only 7 MB, so not worth mentioning. There are some examples which I did not include, because they are not very interesting in the compiled version, but more for understanding the programming part from the source code. OK, I looked into some other packages and tried to split it into pieces (one piece for now). What I do is just add a phase after install and copy things manually (copied from the git package definition): + (add-after 'install 'split + (lambda* (#:key outputs #:allow-other-keys) + (let* ((outputs-out (assoc-ref outputs "out")) + (outputs-feature2d (assoc-ref outputs "feature2d")) + (outputs-face (assoc-ref outputs "face")) + (outputs-core (assoc-ref outputs "core")) + (libface (string-append outputs-out "/lib/libopencv_= face.so")) + (libface* (string-append outputs-face "/lib/libopencv= _face.so")) + (libface3.4 (string-append outputs-out "/lib/libopen= cv_face.so.3.4")) + (libface3.4* (string-append outputs-face "/lib/libope= ncv_face.so.3.4")) + (libface3.4.1 (string-append outputs-out "/lib/libop= encv_face.so.3.4.1")) + (libface3.4.1* (string-append outputs-face "/lib/libo= pencv_face.so.3.4.1"))) + + =20 + (mkdir-p (string-append outputs-face "/lib")) + =20 + (for-each (lambda (old new) + (copy-file old new) + (delete-file old) + (chmod new #o555)) + (list libface libface3.4 libface3.4.1) + (list libface* libface3.4* libface3.4.1*)) + ) + #t ; TODO: Implement it. + )) + ))) The problem here is that this doesn't correct the RPATHS and I will get problems with dependent library parts (The python modules in that case): phase `strip' succeeded after 0.5 seconds starting phase `validate-runpath' validating RUNPATH of 46 binaries in "/gnu/store/f7aqk1my2bdprhgp3gxmnl9227= gxf43m-opencv-3.4.1/lib"... /gnu/store/f7aqk1my2bdprhgp3gxmnl9227gxf43m-opencv-3.4.1/lib/python3.6/site= -packages/cv2.cpython-36m-x86_64-linux-gnu.so: error: depends on 'libopencv= _face.so.3.4', which cannot be found in RUNPATH ("/gnu/store/f7aqk1my2bdprh= gp3gxmnl9227gxf43m-opencv-3.4.1/lib" "/gnu/store/124ymrzp0dwx6qfh4r4r4763sa= 5k48sv-libsm-1.2.2/lib" "/gnu/store/dbdjmralkrzqn6b093hp69bjljvfr7zm-libice= -1.0.9/lib" "/gnu/store/g7sak8qzk7lk06ggn38xpfv5mb8da6kk-libxt-1.1.5/lib" "= /gnu/store/n6acaivs0jwiwpidjr551dhdni5kgpcr-glibc-2.26.105-g0890d5379c/lib"= "/gnu/store/xfjba1kww8ngdc6nxldd8ly93nh13ayy-gcc-5.5.0-lib/lib" "/gnu/stor= e/xfjba1kww8ngdc6nxldd8ly93nh13ayy-gcc-5.5.0-lib/lib/gcc/x86_64-unknown-lin= ux-gnu/5.5.0/../../..") validating RUNPATH of 7 binaries in "/gnu/store/f7aqk1my2bdprhgp3gxmnl9227g= xf43m-opencv-3.4.1/bin"... validating RUNPATH of 3 binaries in "/gnu/store/1356brvn9cjilcp2zizm9gx3vig= mz74i-opencv-3.4.1-face/lib"... phase `validate-runpath' failed after 0.3 seconds So, somehow I have to solve dependencies between the different outputs. I have no idea how to do that. For a :doc output, that doesn't matter, but how can I tell Guix the dependency hierachy of different outputs? =20 > Nitpick: >=20 > > + (synopsis "Computer Vision Library") =20 >=20 > No need to capitalize. OK. =20 > > + (description "OpenCV (Open Source Computer Vision) is a > > library aimed at =20 >=20 > I=E2=80=99d remove the parenthetic part. OK. >=20 > > +real-time computer vision, including several hundred computer > > +vision algorithms.") =20 >=20 > Bonus points if you can expound a little bit, without just listing > those algorithms though. ;-) OK, you got me. I will add more lines when the technical problems are solved. =20 > Apart from that the patch looks very polished and that=E2=80=99s a pleasu= re to > review! Thanks. It took quite some rounds until that point. But good to know you enjoyed the result :-) Bj=C3=B6rn PS: Happy Easter and this is not the xxxx-04-01 fool. --Sig_/mK8Z_wg5wMnG=5wW6+94PrG Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlrACyoACgkQvyhstlk+X/0nggCdE7c9gafKUoxIKtdndOCoCCBP tHMAoJdzQEzzxmn6LBsJt0NOymlL48pL =fPn1 -----END PGP SIGNATURE----- --Sig_/mK8Z_wg5wMnG=5wW6+94PrG--