Re: Tensorflow fixes on core-updates-frozen

[Ricardo Wurmus <rekado@elephly.net>]

Hi,

> [[PGP Signed Part:Undecided]]
> guix-commits@gnu.org skribis:
>
>> rekado pushed a change to branch core-updates-frozen
>> in repository guix.
>>
>>      new d503194  gnu: python2-entrypoints: Add missing input.
>>      new 672c7a2  gnu: tensorflow: Do not unpack directory.
>>      new 6d3439b  gnu: eigen-for-tensorflow: Build with GCC 7.
>
> Hi Ricardo,
>
> Given that the core-update-frozen branch has been merged in master,
> can't these fixes be pushed on master directly?

I accidentally pushed those to core-updates-frozen.  I pushed almost the
same commits to the “master” branch afterwards.

The only difference is in the list of inputs in python2-entrypoints.  It
uses the new style on “master”.

Unfortunately, this is not enough to build tensorflow.  At the very end
we have this problem:

--8<---------------cut here---------------start------------->8---
[ 99%] Building CXX object CMakeFiles/pywrap_tensorflow_internal.dir/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc.o
/gnu/store/vakvgvrb839igv16jkif4lmx11d25jqb-gcc-10.3.0/bin/c++ -DEIGEN_AVOID_STL_ARRAY -DSQLITE_OMIT_LOAD_EXTENSION -DTF_USE_SNAPPY -Dpywrap_tensorflow_internal_EXPORTS -I/tmp/guix-build-tensorflow-1.9.0.drv-0/source -I/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/contrib/build -I/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/contrib/build/external/eigen_archive -I/gnu/store/54dclv4bsbg5894pyh0q5g768113368a-eigen-for-tensorflow-3.3.5-1.fd6845384b86/include/eigen3 -I/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/contrib/build/gemmlowp/src/gemmlowp -I/gnu/store/jh8w9a3gnkisp8zgkklydhc372vgcsk6-jsoncpp-1.7.3 -I/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/contrib/build/external/farmhash_archive -I/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/contrib/build/external/farmhash_archive/util -I/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/contrib/build/external/highwayhash -I/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/contrib/build/cub/src/cub -I/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/contrib/build/external/nsync/public -I/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/contrib/build/re2/install/include -I/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/contrib/build/double_conversion/src/double_conversion -I/gnu/store/g88hslvix5j2rkdm13ay7biqyyqfja0f-snappy-1.1.9 -I/gnu/store/p5fgysbcnnp8b1d91mrvjvababmczga0-python-3.9.6/include/python3.9 -I/gnu/store/wgsmkn68q8h178sqc7ywjcdr330z9rb6-python-numpy-1.20.3/lib/python3.9/site-packages/numpy/core/include -fPIC -fno-exceptions -std=c++11 -fopenmp -O3 -DNDEBUG -fPIC -std=gnu++14 -MD -MT CMakeFiles/pywrap_tensorflow_internal.dir/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc.o -MF CMakeFiles/pywrap_tensorflow_internal.dir/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc.o.d -o CMakeFiles/pywrap_tensorflow_internal.dir/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc.o -c /tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc
/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc:362:1: warning: converting to non-pointer type ‘long int’ from NULL [-Wconversion-null]
  362 | };
      | ^
/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc: In function ‘bool tensorflow::{anonymous}::Initialize()’:
/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc:623:36: error: no match for call to ‘(tensorflow::{anonymous}::Initialize()::<lambda(const char*, PyUFuncGenericFunction, const std::array<int, 3>&)>) (const char [6], <unresolved overloaded function type>, const std::array<int, 3>&)’
  623 |                       compare_types)) {
      |                                    ^
/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc:596:25: note: candidate: ‘tensorflow::{anonymous}::Initialize()::<lambda(const char*, PyUFuncGenericFunction, const std::array<int, 3>&)>’
  596 |   auto register_ufunc = [&](const char* name, PyUFuncGenericFunction fn,
      |                         ^
/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc:596:25: note:   no known conversion for argument 2 from ‘<unresolved overloaded function type>’ to ‘PyUFuncGenericFunction’ {aka ‘void (*)(char**, const long int*, const long int*, void*)’}
/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc:627:36: error: no match for call to ‘(tensorflow::{anonymous}::Initialize()::<lambda(const char*, PyUFuncGenericFunction, const std::array<int, 3>&)>) (const char [10], <unresolved overloaded function type>, const std::array<int, 3>&)’
  627 |                       compare_types)) {
      |                                    ^
/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc:596:25: note: candidate: ‘tensorflow::{anonymous}::Initialize()::<lambda(const char*, PyUFuncGenericFunction, const std::array<int, 3>&)>’
  596 |   auto register_ufunc = [&](const char* name, PyUFuncGenericFunction fn,
      |                         ^
/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc:596:25: note:   no known conversion for argument 2 from ‘<unresolved overloaded function type>’ to ‘PyUFuncGenericFunction’ {aka ‘void (*)(char**, const long int*, const long int*, void*)’}
/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc:630:77: error: no match for call to ‘(tensorflow::{anonymous}::Initialize()::<lambda(const char*, PyUFuncGenericFunction, const std::array<int, 3>&)>) (const char [5], <unresolved overloaded function type>, const std::array<int, 3>&)’
  630 |   if (!register_ufunc("less", CompareUFunc<Bfloat16LtFunctor>, compare_types)) {
      |                                                                             ^
/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc:596:25: note: candidate: ‘tensorflow::{anonymous}::Initialize()::<lambda(const char*, PyUFuncGenericFunction, const std::array<int, 3>&)>’
  596 |   auto register_ufunc = [&](const char* name, PyUFuncGenericFunction fn,
      |                         ^
/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc:596:25: note:   no known conversion for argument 2 from ‘<unresolved overloaded function type>’ to ‘PyUFuncGenericFunction’ {aka ‘void (*)(char**, const long int*, const long int*, void*)’}
/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc:634:36: error: no match for call to ‘(tensorflow::{anonymous}::Initialize()::<lambda(const char*, PyUFuncGenericFunction, const std::array<int, 3>&)>) (const char [8], <unresolved overloaded function type>, const std::array<int, 3>&)’
  634 |                       compare_types)) {
      |                                    ^
/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc:596:25: note: candidate: ‘tensorflow::{anonymous}::Initialize()::<lambda(const char*, PyUFuncGenericFunction, const std::array<int, 3>&)>’
  596 |   auto register_ufunc = [&](const char* name, PyUFuncGenericFunction fn,
      |                         ^
/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc:596:25: note:   no known conversion for argument 2 from ‘<unresolved overloaded function type>’ to ‘PyUFuncGenericFunction’ {aka ‘void (*)(char**, const long int*, const long int*, void*)’}
/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc:638:36: error: no match for call to ‘(tensorflow::{anonymous}::Initialize()::<lambda(const char*, PyUFuncGenericFunction, const std::array<int, 3>&)>) (const char [11], <unresolved overloaded function type>, const std::array<int, 3>&)’
  638 |                       compare_types)) {
      |                                    ^
/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc:596:25: note: candidate: ‘tensorflow::{anonymous}::Initialize()::<lambda(const char*, PyUFuncGenericFunction, const std::array<int, 3>&)>’
  596 |   auto register_ufunc = [&](const char* name, PyUFuncGenericFunction fn,
      |                         ^
/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc:596:25: note:   no known conversion for argument 2 from ‘<unresolved overloaded function type>’ to ‘PyUFuncGenericFunction’ {aka ‘void (*)(char**, const long int*, const long int*, void*)’}
/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc:642:36: error: no match for call to ‘(tensorflow::{anonymous}::Initialize()::<lambda(const char*, PyUFuncGenericFunction, const std::array<int, 3>&)>) (const char [14], <unresolved overloaded function type>, const std::array<int, 3>&)’
  642 |                       compare_types)) {
      |                                    ^
/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc:596:25: note: candidate: ‘tensorflow::{anonymous}::Initialize()::<lambda(const char*, PyUFuncGenericFunction, const std::array<int, 3>&)>’
  596 |   auto register_ufunc = [&](const char* name, PyUFuncGenericFunction fn,
      |                         ^
/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc:596:25: note:   no known conversion for argument 2 from ‘<unresolved overloaded function type>’ to ‘PyUFuncGenericFunction’ {aka ‘void (*)(char**, const long int*, const long int*, void*)’}
make[2]: *** [CMakeFiles/pywrap_tensorflow_internal.dir/build.make:188: CMakeFiles/pywrap_tensorflow_internal.dir/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/python/lib/core/bfloat16.cc.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory '/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/contrib/build'
make[1]: *** [CMakeFiles/Makefile2:3807: CMakeFiles/pywrap_tensorflow_internal.dir/all] Error 2
make[1]: Leaving directory '/tmp/guix-build-tensorflow-1.9.0.drv-0/source/tensorflow/contrib/build'
make: *** [Makefile:139: all] Error 2
--8<---------------cut here---------------end--------------->8---

That’s because Tensorflow 1.9.0 is not meant to be built with Python
3.9.  It didn’t even like Python 3.6, which is why we already carry a
few 3.7 compatibility patches.  We should figure out how the calling
convention changed and then patch the C++ files.

My changes compared to the “master” branch:

--8<---------------cut here---------------start------------->8---
diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index 58f7c7c013..9d8bed09f7 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -1901,6 +1901,9 @@ (define-public tensorflow
              (mkdir-p "eigen-src")
              (copy-recursively (assoc-ref inputs "eigen:src") "eigen-src")
 
+             (substitute* "tensorflow/tools/ci_build/install/install_pip_packages.sh"
+               (("setuptools==39.1.0") "setuptools"))
+
              (substitute* "tensorflow/contrib/cmake/tf_python.cmake"
                ;; Take protobuf source files from our source package.
                (("\\$\\{CMAKE_CURRENT_BINARY_DIR\\}/protobuf/src/protobuf/src/google")
@@ -1957,8 +1960,7 @@ (define-public tensorflow
        ("protobuf:native" ,protobuf-3.6) ; protoc
        ("protobuf:src" ,(package-source protobuf-3.6))
        ("eigen:src" ,(package-source eigen-for-tensorflow))
-       ;; install_pip_packages.sh wants setuptools 39.1.0 specifically.
-       ("python-setuptools" ,python-setuptools-for-tensorflow)
+       ("python-setuptools" ,python-setuptools)
 
        ;; The commit hashes and URLs for third-party source code are taken
        ;; from "tensorflow/workspace.bzl".
--8<---------------cut here---------------end--------------->8---


-- 
Ricardo