[bug#31386] TensorFlow

[bug#31386] TensorFlow

[Ricardo Wurmus]

This patch series adds a package for TensorFlow (without GPU support).
I don’t know how useful this is because it just provides a single 180+MB
statically linked archive (and header files).  It is only the “core”
part of TensorFlow.

The problem here is that the build system is Bazel, which cannot easily
be built from source.  Bazel bundles 60+ third-party Java packages, some
of which are very complex and are very difficult to package.  TensorFlow
comes with an unsupported Makefile that builds a single statically
linked library of only the core parts.  That’s what this package
provides.

Note that TensorFlow 1.8 has been released a few days ago.  This package
only adds version 1.7.

--
Ricardo

[bug#31386] [PATCH 1/3] gnu: eigen: Include unsupported features.

[Ricardo Wurmus]

* gnu/packages/algebra.scm (eigen)[source]: Do not build the tests for
unsupported features, but include the files.
---
 gnu/packages/algebra.scm | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/algebra.scm b/gnu/packages/algebra.scm
index 03a61be47..6daf588cd 100644
--- a/gnu/packages/algebra.scm
+++ b/gnu/packages/algebra.scm
@@ -732,9 +732,10 @@ Sine Transform} (DST) and @dfn{Discrete Hartley Transform} (DHT).")
                ;; but maintainers say it's a known issue and it's unsupported
                ;; anyway, so just skip them.
                '(begin
+                  (substitute* "unsupported/CMakeLists.txt"
+                    (("add_subdirectory\\(test.*")
+                     "# Do not build the tests for unsupported features.\n"))
 		  (substitute* "CMakeLists.txt"
-                    (("add_subdirectory\\(unsupported\\)")
-                     "# Do not build the tests for unsupported features.\n")
                     ;; Work around
                     ;; <http://eigen.tuxfamily.org/bz/show_bug.cgi?id=1114>.
                     (("\"include/eigen3\"")
-- 
2.15.1

[bug#31386] [PATCH 2/3] gnu: Add gemmlowp-for-tensorflow.

[Ricardo Wurmus]

* gnu/packages/machine-learning.scm (gemmlowp-for-tensorflow): New variable.
---
 gnu/packages/machine-learning.scm | 54 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 53 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index f0d35484e..87fbec6b6 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2015, 2016, 2017 Ricardo Wurmus <rekado@elephly.net>
+;;; Copyright © 2015, 2016, 2017, 2018 Ricardo Wurmus <rekado@elephly.net>
 ;;; Copyright © 2016 Efraim Flashner <efraim@flashner.co.il>
 ;;; Copyright © 2016, 2017 Marius Bakke <mbakke@fastmail.com>
 ;;; Copyright © 2016 Hartmut Goebel <h.goebel@crazy-compilers.com>
@@ -557,6 +557,58 @@ Support Vector Machines, Spectral Clustering, Kernel PCA, Gaussian Processes
 and a QP solver.")
     (license license:gpl2)))
 
+;; TODO: We may need to pass "-msse4.1" as an additional optimization flag
+;; when building for x86_64.  This is strongly recommended by the authors in
+;; the README.md.
+(define-public gemmlowp-for-tensorflow
+  (let ((commit "7c7c744640ddc3d0af18fb245b4d23228813a71b"))
+    (package
+      (name "gemmlowp")
+      (version (string-append "0-1." (string-take commit 7)))
+      (source (origin
+                (method url-fetch)
+                (uri (string-append "https://mirror.bazel.build/"
+                                    "github.com/google/gemmlowp/archive/"
+                                    commit ".zip"))
+                (sha256
+                 (base32
+                  "1kxj8h017q4r1dcva61vziwmks66rvr0iw93lg45fwws4n8cqlmq"))))
+      (build-system cmake-build-system)
+      (arguments
+       `(#:phases
+         (modify-phases %standard-phases
+           ;; This directory contains the CMakeLists.txt.
+           (add-after 'unpack 'chdir
+             (lambda _ (chdir "contrib") #t))
+           ;; There is no install target
+           (replace 'install
+             (lambda* (#:key outputs #:allow-other-keys)
+               (let* ((out (assoc-ref outputs "out"))
+                      (lib (string-append out "/lib/"))
+                      (inc (string-append out "/include/")))
+                 (install-file "../build/libeight_bit_int_gemm.so" lib)
+                 (for-each (lambda (dir)
+                             (let ((target (string-append inc "/" dir)))
+                               (mkdir-p target)
+                               (for-each (lambda (h)
+                                           (install-file h target))
+                                         (find-files (string-append "../" dir)
+                                                     "\\.h$"))))
+                           '("meta" "profiling" "public" "fixedpoint"
+                             "eight_bit_int_gemm" "internal"))
+                 #t))))))
+      (native-inputs
+       `(("unzip" ,unzip)))
+      (home-page "https://github.com/google/gemmlowp")
+      (synopsis "Small self-contained low-precision GEMM library")
+      (description
+       "This is a small self-contained low-precision @dfn{general matrix
+multiplication} (GEMM) library.  It is not a full linear algebra library.
+Low-precision means that the input and output matrix entries are integers on
+at most 8 bits.  To avoid overflow, results are internally accumulated on more
+than 8 bits, and at the end only some significant 8 bits are kept.")
+      (license license:asl2.0))))
+
 (define-public dlib
   (package
     (name "dlib")
-- 
2.15.1

[bug#31386] [PATCH 2/3] gnu: Add gemmlowp-for-tensorflow.

[Ludovic Courtès]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> * gnu/packages/machine-learning.scm (gemmlowp-for-tensorflow): New variable.

[...]

> +;; TODO: We may need to pass "-msse4.1" as an additional optimization flag
> +;; when building for x86_64.  This is strongly recommended by the authors in
> +;; the README.md.

It’d be nice to produce a patch to have “function multi-versioning” for
the hot parts, as described at
<https://clearlinux.org/documentation/clear-linux/tutorials/fmv>.

> +(define-public gemmlowp-for-tensorflow
> +  (let ((commit "7c7c744640ddc3d0af18fb245b4d23228813a71b"))

Could you comment on how you chose this commit?  Is it because it starts
with “7c7c”?  ;-)

Otherwise LGTM!

Ludo’.

[bug#31386] [PATCH 2/3] gnu: Add gemmlowp-for-tensorflow.

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:
>
>> * gnu/packages/machine-learning.scm (gemmlowp-for-tensorflow): New variable.
>
> [...]
>
>> +;; TODO: We may need to pass "-msse4.1" as an additional optimization flag
>> +;; when building for x86_64.  This is strongly recommended by the authors in
>> +;; the README.md.
>
> It’d be nice to produce a patch to have “function multi-versioning” for
> the hot parts, as described at
> <https://clearlinux.org/documentation/clear-linux/tutorials/fmv>.

Yeah, that would be nice.  However, I don’t know how to test the
resulting binary.

>> +(define-public gemmlowp-for-tensorflow
>> +  (let ((commit "7c7c744640ddc3d0af18fb245b4d23228813a71b"))
>
> Could you comment on how you chose this commit?  Is it because it starts
> with “7c7c”?  ;-)

That’s the commit that the TensorFlow build system refers to.

--
Ricardo

[bug#31386] [PATCH 3/3] gnu: Add tensorflow-core.

[Ricardo Wurmus]

* gnu/packages/machine-learning.scm (tensorflow-core): New variable.
---
 gnu/packages/machine-learning.scm | 113 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 113 insertions(+)

diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index 87fbec6b6..e02e89ef4 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -36,6 +36,7 @@
   #:use-module (guix build-system r)
   #:use-module (guix git-download)
   #:use-module (gnu packages)
+  #:use-module (gnu packages algebra)
   #:use-module (gnu packages autotools)
   #:use-module (gnu packages boost)
   #:use-module (gnu packages check)
@@ -48,6 +49,7 @@
   #:use-module (gnu packages ocaml)
   #:use-module (gnu packages perl)
   #:use-module (gnu packages pkg-config)
+  #:use-module (gnu packages protobuf)
   #:use-module (gnu packages python)
   #:use-module (gnu packages statistics)
   #:use-module (gnu packages swig)
@@ -609,6 +611,117 @@ at most 8 bits.  To avoid overflow, results are internally accumulated on more
 than 8 bits, and at the end only some significant 8 bits are kept.")
       (license license:asl2.0))))
 
+(define-public tensorflow-core
+  (package
+    (name "tensorflow-core")
+    (version "1.7.0")
+    (source
+     (origin
+       (method git-fetch)
+       (uri (git-reference
+             (url "https://github.com/tensorflow/tensorflow.git")
+             (commit (string-append "v" version))))
+       (file-name (string-append "tensorflow-" version "-checkout"))
+       (sha256
+        (base32
+         "0jljzbwhmxi8crbivwachcmlfrrv279qrsvwc62cnnbyw0n1g0kp"))))
+    (build-system gnu-build-system)
+    (arguments
+     `(#:tests? #f ; no "check" target
+       #:make-flags
+       (list "-f" "tensorflow/contrib/makefile/Makefile"
+             "CC=gcc")
+       #:phases
+       (modify-phases %standard-phases
+         (delete 'configure)
+         (add-after 'unpack 'fix-version
+           (lambda _
+             (substitute* "tensorflow/tools/git/gen_git_source.sh"
+               (("^GIT_VERSION=.*")
+                (string-append "GIT_VERSION=" ,version "\n")))
+             #t))
+         (add-after 'unpack 'unpack-third-party
+           (lambda* (#:key inputs #:allow-other-keys)
+             (with-directory-excursion "tensorflow/contrib/makefile/"
+               (let ((fft2d "downloads/fft2d")
+                     (nsync "downloads/nsync"))
+                 (mkdir-p fft2d)
+                 (invoke "tar" "xf" (assoc-ref inputs "fft2d")
+                         "-C" fft2d "--strip-components=1")
+                 (mkdir-p nsync)
+                 (invoke "tar" "xf" (assoc-ref inputs "nsync")
+                         "-C" nsync "--strip-components=1")))))
+         ;; FIXME: it would be nice to build a separate package for nsync and
+         ;; use it here.  Unfortunately, I could not build Tensorflow with a
+         ;; separately built nsync.
+         (add-before 'build 'build-nsync
+           (lambda _
+             (with-directory-excursion "tensorflow/contrib/makefile/"
+               (invoke "bash" "compile_nsync.sh")
+               (setenv "TARGET_NSYNC_LIB"
+                       "tensorflow/contrib/makefile/downloads/nsync/builds/default.linux.c++11/nsync.a")
+               (setenv "HOST_NSYNC_LIB"
+                       "tensorflow/contrib/makefile/downloads/nsync/builds/default.linux.c++11/nsync.a")
+               #t)))
+         (add-after 'unpack 'find-eigen-headers
+           (lambda* (#:key inputs #:allow-other-keys)
+             ;; Ensure that Eigen headers can be found
+             (setenv "CPLUS_INCLUDE_PATH"
+                     (string-append (getenv "CPLUS_INCLUDE_PATH")
+                                    ":"
+                                    (assoc-ref inputs "eigen")
+                                    "/include/eigen3"))
+             #t))
+         (replace 'install
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (lib (string-append out "/lib"))
+                    (inc (string-append out "/include")))
+               (install-file "tensorflow/contrib/makefile/gen/lib/libtensorflow-core.a" lib)
+               (for-each (lambda (file)
+                           (let ((target (string-append inc "/"
+                                                        (dirname file))))
+                             (mkdir-p target)
+                             (install-file file target)))
+                         (find-files "tensorflow/core" ".*\\.h$"))
+               #t))))))
+    (native-inputs
+     `(("protobuf" ,protobuf)           ; protoc
+       ;; "You may use, copy, modify this code for any purpose and without
+       ;; fee. You may distribute this ORIGINAL package."
+       ("fft2d"
+        ,(origin
+           (method url-fetch)
+           (uri "https://mirror.bazel.build/www.kurims.kyoto-u.ac.jp/~ooura/fft.tgz")
+           (sha256
+            (base32
+             "15jjkfvhqvl2c0753d2di8hz0pyzn598g74wqy79awdrf1y67fsj"))))
+       ("nsync"
+        ,(origin
+           (method url-fetch)
+           (uri (string-append "https://mirror.bazel.build/"
+                               "github.com/google/nsync/archive/"
+                               "0559ce013feac8db639ee1bf776aca0325d28777.tar.gz"))
+           (sha256
+            (base32
+             "0qdkyqym34x739mmzv97ah5r7ph462v5xkxqxvidmcfqbi64b132"))))
+       ("googletest" ,googletest)))
+    (inputs
+     `(("eigen" ,eigen-for-tensorflow)
+       ("gemmlowp" ,gemmlowp-for-tensorflow)
+       ("protobuf" ,protobuf)
+       ("zlib" ,zlib)))
+    (home-page "https://tensorflow.org")
+    (synopsis "Machine learning framework")
+    (description
+     "TensorFlow is a software library for high performance numerical
+computation.  Its flexible architecture allows easy deployment of computation
+across a variety of platforms, and from desktops to clusters of servers to
+mobile and edge devices.
+
+This package provides only the core library.")
+    (license license:asl2.0)))
+
 (define-public dlib
   (package
     (name "dlib")
-- 
2.15.1

[bug#31386] [PATCH 3/3] gnu: Add tensorflow-core.

[Ludovic Courtès]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> * gnu/packages/machine-learning.scm (tensorflow-core): New variable.

[...]

> +       #:make-flags
> +       (list "-f" "tensorflow/contrib/makefile/Makefile"
> +             "CC=gcc")

Perhaps we should add a comment on why not Bazel.

> +    (native-inputs
> +     `(("protobuf" ,protobuf)           ; protoc
> +       ;; "You may use, copy, modify this code for any purpose and without
> +       ;; fee. You may distribute this ORIGINAL package."
> +       ("fft2d"

Doesn’t that make fft2d non-free, if we cannot redistribute modified copies?

> +        ,(origin
> +           (method url-fetch)
> +           (uri "https://mirror.bazel.build/www.kurims.kyoto-u.ac.jp/~ooura/fft.tgz")

Unversioned URL.  :-/

Thank you!

Ludo’.

[bug#31386] [PATCH 3/3] gnu: Add tensorflow-core.

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:
>
>> * gnu/packages/machine-learning.scm (tensorflow-core): New variable.
>
> [...]
>
>> +       #:make-flags
>> +       (list "-f" "tensorflow/contrib/makefile/Makefile"
>> +             "CC=gcc")
>
> Perhaps we should add a comment on why not Bazel.
>
>> +    (native-inputs
>> +     `(("protobuf" ,protobuf)           ; protoc
>> +       ;; "You may use, copy, modify this code for any purpose and without
>> +       ;; fee. You may distribute this ORIGINAL package."
>> +       ("fft2d"
>
> Doesn’t that make fft2d non-free, if we cannot redistribute modified copies?

I really don’t understand the second sentence.  The first sentence says
that “this code” may be copied and modified “for any purpose”.  The
second sentence doesn’t seem to add anything, but that depends on how
you read it.

   You may *only* distribute this [source archive].
   You may *also* distribute this [unmodified package].

I wrote an email to the author’s email address, but I don’t know if that
address is still in use.

Anyway: I just went to http://www.kurims.kyoto-u.ac.jp/~ooura/fft.html,
where this sentence has been removed.  The license notice now looks like
this:

    Copyright Takuya OOURA, 1996-2001

    You may use, copy, modify and distribute this code for any purpose
    (include commercial use) and without fee. Please refer to this
    package when you modify this code.

The notice in the readme.txt is unchanged, though.

>> +        ,(origin
>> +           (method url-fetch)
>> +           (uri "https://mirror.bazel.build/www.kurims.kyoto-u.ac.jp/~ooura/fft.tgz")
>
> Unversioned URL.  :-/

Yeah :(

The upstream URL is also unversioned, but it looks like development has
ceased in 2006.

--
Ricardo

[bug#31386] [PATCH 3/3] gnu: Add tensorflow-core.

[Ludovic Courtès]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> Anyway: I just went to http://www.kurims.kyoto-u.ac.jp/~ooura/fft.html,
> where this sentence has been removed.  The license notice now looks like
> this:
>
>     Copyright Takuya OOURA, 1996-2001
>
>     You may use, copy, modify and distribute this code for any purpose
>     (include commercial use) and without fee. Please refer to this
>     package when you modify this code.

That looks better, thanks for checking.

Ludo’.

[bug#31386] [PATCH 1/3] gnu: eigen: Include unsupported features.

[Ludovic Courtès]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> * gnu/packages/algebra.scm (eigen)[source]: Do not build the tests for
> unsupported features, but include the files.

I was reluctant about installing those, but IIRC they go to
$includedir/…/unsupported, right?  In that case, it’s up to users to
explicitly #include from unsupported/ so it’s probably fine.

Thanks,
Ludo’.

[bug#31386] [PATCH 1/3] gnu: eigen: Include unsupported features.

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:
>
>> * gnu/packages/algebra.scm (eigen)[source]: Do not build the tests for
>> unsupported features, but include the files.
>
> I was reluctant about installing those, but IIRC they go to
> $includedir/…/unsupported, right?  In that case, it’s up to users to
> explicitly #include from unsupported/ so it’s probably fine.

That’s correct.

As it turns out, this alone is not sufficient for TensorFlow.  They have
some additional Eigen source files that they bundle with the TensorFlow
sources, which must be present at build time.

--
Ricardo

[bug#31386] TensorFlow

[Fis Trivial]

Ricardo Wurmus writes:

> This patch series adds a package for TensorFlow (without GPU support).
> I don’t know how useful this is because it just provides a single 180+MB
> statically linked archive (and header files).  It is only the “core”
> part of TensorFlow.
>
> The problem here is that the build system is Bazel, which cannot easily
> be built from source.  Bazel bundles 60+ third-party Java packages, some
> of which are very complex and are very difficult to package.  TensorFlow
> comes with an unsupported Makefile that builds a single statically
> linked library of only the core parts.  That’s what this package
> provides.
>
> Note that TensorFlow 1.8 has been released a few days ago.  This package
> only adds version 1.7.

That's still amazing.

I thought about packaging bazel and looked into the nix package[1]. The
package definition from nix isn't very long, but requires patches. I
dropped the matter since I'm not familiar with java.

[1]: https://github.com/NixOS/nixpkgs/tree/master/pkgs/development/tools/build-managers/bazel

[bug#31386] TensorFlow

[Björn Höfling]

[-- Attachment #1: Type: text/plain, Size: 630 bytes --]

On Tue, 8 May 2018 17:52:10 +0000
Fis Trivial <ybbs.daans@hotmail.com> wrote:


> I thought about packaging bazel and looked into the nix package[1].
> The package definition from nix isn't very long, but requires
> patches. I dropped the matter since I'm not familiar with java.
> 
> [1]:
> https://github.com/NixOS/nixpkgs/tree/master/pkgs/development/tools/build-managers/bazel

NixOS is just using the bundled dependencies. For Guix, we at least
have to check that all this bundled source code is really free
software, not just open source. Better we unbundle them and use the
dependencies from Guix.

Björn

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[bug#31386] TensorFlow

[Ludovic Courtès]

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> This patch series adds a package for TensorFlow (without GPU support).
> I don’t know how useful this is because it just provides a single 180+MB
> statically linked archive (and header files).  It is only the “core”
> part of TensorFlow.

Quite an achievement!  I wonder if people would normally use this .a
library.  Doesn’t TensorFlow have Python bindings as well?

Thank you,
Ludo’.

[bug#31386] TensorFlow

[Ricardo Wurmus]

The new patches can be seen here:

    https://issues.guix.info/issue/33952

-- 
Ricardo