Request: Bazel build system (required for Tensorflow update)

Request: Bazel build system (required for Tensorflow update)

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 219 bytes --]

Hi,

Has anyone worked on a Bazel build system?
https://docs.bazel.build/versions/master/install.html

It's required for Tensorflow somewhere between 1.10 and 1.14.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Request: Bazel build system (required for Tensorflow update)

[Julien Lepiller]

Le 4 mars 2020 05:12:00 GMT-05:00, Pierre Neidhardt <mail@ambrevar.xyz> a écrit :
>Hi,
>
>Has anyone worked on a Bazel build system?
>https://docs.bazel.build/versions/master/install.html
>
>It's required for Tensorflow somewhere between 1.10 and 1.14.

I and Ricardo have looked at it in the past, and it has a huge dependency graph. Their pinning of dependencies also makes it a bit difficult to work with, but we can probably find a workaround.

Re: Request: Bazel build system (required for Tensorflow update)

[Roel Janssen]

On Wed, 2020-03-04 at 06:56 -0500, Julien Lepiller wrote:
> Le 4 mars 2020 05:12:00 GMT-05:00, Pierre Neidhardt <mail@ambrevar.xyz> a
> écrit :
> > Hi,
> > 
> > Has anyone worked on a Bazel build system?
> > https://docs.bazel.build/versions/master/install.html
> > 
> > It's required for Tensorflow somewhere between 1.10 and 1.14.
> 
> I and Ricardo have looked at it in the past, and it has a huge dependency
> graph. Their pinning of dependencies also makes it a bit difficult to work
> with, but we can probably find a workaround.

AFAICS does tensorflow 1.15.2 still have the CMake build system files in place.

What has changed since 1.9.0 that makes the CMake build system no option
anymore?

Kind regards,
Roel Janssen

Re: Request: Bazel build system (required for Tensorflow update)

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 5355 bytes --]

Maybe CMake can still be used.

I just packaged abseil-cpp and added it to the inputs of tensorflow
1.15.2.
I get the following error though:

--8<---------------cut here---------------start------------->8---
starting phase `configure'
source directory: "/tmp/guix-build-tensorflow-1.15.0.drv-0/source/tensorflow/contrib/cmake" (relative from build: "../cmake")
build directory: "/tmp/guix-build-tensorflow-1.15.0.drv-0/source/tensorflow/contrib/build"
running 'cmake' with arguments ("../cmake" "-DCMAKE_BUILD_TYPE=Release" "-DCMAKE_INSTALL_PREFIX=/gnu/store/6bjfms9h8p1v42ai71wyvrh3bwxyzl1y-tensorflow-1.15.0" "-DCMAKE_INSTALL_LIBDIR=lib" "-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=TRUE" "-DCMAKE_INSTALL_RPATH=/gnu/store/6bjfms9h8p1v42ai71wyvrh3bwxyzl1y-tensorflow-1.15.0/lib" "-DCMAKE_VERBOSE_MAKEFILE=ON" "-Dprotobuf_STATIC_LIBRARIES=/gnu/store/p77n8kpsl50qlrz5fk0mc9kvfkinh9dq-protobuf-3.6.1/lib/libprotobuf.so" "-DPROTOBUF_PROTOC_EXECUTABLE=/gnu/store/p77n8kpsl50qlrz5fk0mc9kvfkinh9dq-protobuf-3.6.1/bin/protoc" "-Dsnappy_STATIC_LIBRARIES=/gnu/store/3xrpbdhhb8nk9p9jqr19ljlyhxnxk18n-snappy-1.1.8/lib/libsnappy.so" "-Dsnappy_INCLUDE_DIR=/gnu/store/3xrpbdhhb8nk9p9jqr19ljlyhxnxk18n-snappy-1.1.8" "-Djsoncpp_STATIC_LIBRARIES=/gnu/store/3vacq5lrnri4g8a5498qxkgxv4z8jyv8-jsoncpp-1.7.3/lib/libjsoncpp.so" "-Djsoncpp_INCLUDE_DIR=/gnu/store/3vacq5lrnri4g8a5498qxkgxv4z8jyv8-jsoncpp-1.7.3" "-Dsqlite_STATIC_LIBRARIES=/gnu/store/i6l1579g80387rda658jy9cfqq82643d-sqlite-3.28.0/lib/libsqlite.a" "-DABSEIL_CPP_LIBRARIES=/gnu/store/j36v4bb2vzy02xfyxp6ryfm4aj0yiaxn-abseil-cpp-20200225/lib/" "-Dsystemlib_ALL=ON" "-Dtensorflow_ENABLE_POSITION_INDEPENDENT_CODE=ON" "-Dtensorflow_BUILD_SHARED_LIB=ON" "-Dtensorflow_OPTIMIZE_FOR_NATIVE_ARCH=OFF" "-Dtensorflow_ENABLE_SSL_SUPPORT=OFF" "-Dtensorflow_BUILD_CONTRIB_KERNELS=OFF")
-- The C compiler identification is GNU 7.4.0
-- The CXX compiler identification is GNU 7.4.0
-- Check for working C compiler: /gnu/store/x3jx25cd3q363mr7nbgzrhrv1vza6cf7-gcc-7.4.0/bin/gcc
-- Check for working C compiler: /gnu/store/x3jx25cd3q363mr7nbgzrhrv1vza6cf7-gcc-7.4.0/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /gnu/store/x3jx25cd3q363mr7nbgzrhrv1vza6cf7-gcc-7.4.0/bin/c++
-- Check for working CXX compiler: /gnu/store/x3jx25cd3q363mr7nbgzrhrv1vza6cf7-gcc-7.4.0/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Performing Test GCC_OPENMP_SUPPORT
-- Performing Test GCC_OPENMP_SUPPORT - Success
-- Found PkgConfig: /gnu/store/i1cqaixp79vd3qwnyj1ll10pq6skm2wk-pkg-config-0.29.2/bin/pkg-config (found version "0.29.2") 
-- Checking for one of the modules 'zlib'
CMake Error at modules/FindAbseilCpp.cmake:47 (message):
  

  abseil_cpp library "str_format_internal" not found in system path.

  Please provide locations using:
  -DABSEIL_CPP_LIBRARIES_DIR_HINTS:STRING="PATH"

Call Stack (most recent call first):
  external/abseil_cpp.cmake:17 (find_package)
  CMakeLists.txt:264 (include)


-- Configuring incomplete, errors occurred!
See also "/tmp/guix-build-tensorflow-1.15.0.drv-0/source/tensorflow/contrib/build/CMakeFiles/CMakeOutput.log".
See also "/tmp/guix-build-tensorflow-1.15.0.drv-0/source/tensorflow/contrib/build/CMakeFiles/CMakeError.log".
command "cmake" "../cmake" "-DCMAKE_BUILD_TYPE=Release" "-DCMAKE_INSTALL_PREFIX=/gnu/store/6bjfms9h8p1v42ai71wyvrh3bwxyzl1y-tensorflow-1.15.0" "-DCMAKE_INSTALL_LIBDIR=lib" "-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=TRUE" "-DCMAKE_INSTALL_RPATH=/gnu/store/6bjfms9h8p1v42ai71wyvrh3bwxyzl1y-tensorflow-1.15.0/lib" "-DCMAKE_VERBOSE_MAKEFILE=ON" "-Dprotobuf_STATIC_LIBRARIES=/gnu/store/p77n8kpsl50qlrz5fk0mc9kvfkinh9dq-protobuf-3.6.1/lib/libprotobuf.so" "-DPROTOBUF_PROTOC_EXECUTABLE=/gnu/store/p77n8kpsl50qlrz5fk0mc9kvfkinh9dq-protobuf-3.6.1/bin/protoc" "-Dsnappy_STATIC_LIBRARIES=/gnu/store/3xrpbdhhb8nk9p9jqr19ljlyhxnxk18n-snappy-1.1.8/lib/libsnappy.so" "-Dsnappy_INCLUDE_DIR=/gnu/store/3xrpbdhhb8nk9p9jqr19ljlyhxnxk18n-snappy-1.1.8" "-Djsoncpp_STATIC_LIBRARIES=/gnu/store/3vacq5lrnri4g8a5498qxkgxv4z8jyv8-jsoncpp-1.7.3/lib/libjsoncpp.so" "-Djsoncpp_INCLUDE_DIR=/gnu/store/3vacq5lrnri4g8a5498qxkgxv4z8jyv8-jsoncpp-1.7.3" "-Dsqlite_STATIC_LIBRARIES=/gnu/store/i6l1579g80387rda658jy9cfqq82643d-sqlite-3.28.0/lib/libsqlite.a" "-DABSEIL_CPP_LIBRARIES=/gnu/store/j36v4bb2vzy02xfyxp6ryfm4aj0yiaxn-abseil-cpp-20200225/lib/" "-Dsystemlib_ALL=ON" "-Dtensorflow_ENABLE_POSITION_INDEPENDENT_CODE=ON" "-Dtensorflow_BUILD_SHARED_LIB=ON" "-Dtensorflow_OPTIMIZE_FOR_NATIVE_ARCH=OFF" "-Dtensorflow_ENABLE_SSL_SUPPORT=OFF" "-Dtensorflow_BUILD_CONTRIB_KERNELS=OFF" failed with status 1
--8<---------------cut here---------------end--------------->8---

Any clue?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Request: Bazel build system (required for Tensorflow update)

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 3931 bytes --]

I've actually managed to fix the abseil issue.

Now I'm stuck at more broken CMake stuff.
Duh, this will take a while...

--8<---------------cut here---------------start------------->8---
-- Could NOT find c-ares (missing: c-ares_DIR)
-- Found PythonInterp: /gnu/store/l8nphg0idd8pfddyad8f92lx8d1hc053-python-wrapper-3.7.4/bin/python (found version "3.7.4") 
-- Found PythonLibs: /gnu/store/78w7y0lxar70j512iqw8x3nimzj10yga-python-3.7.4/lib/libpython3.7m.so (found version "3.7.4") 
CMake Error at tf_python.cmake:132 (message):
  Python proto directory not found: tensorflow/contrib/tpu/profiler
Call Stack (most recent call first):
  CMakeLists.txt:613 (include)


CMake Error at tf_python.cmake:217 (message):
  Python module not found: tensorflow/contrib/rnn/kernels
Call Stack (most recent call first):
  CMakeLists.txt:613 (include)


CMake Error at tf_python.cmake:217 (message):
  Python module not found: tensorflow/contrib/rnn/ops
Call Stack (most recent call first):
  CMakeLists.txt:613 (include)


CMake Error at tf_python.cmake:217 (message):
  Python module not found: tensorflow/contrib/tpu/profiler
Call Stack (most recent call first):
  CMakeLists.txt:613 (include)


-- Found SWIG: /gnu/store/1jamhp01xc911m68j8ndiwlcc55q8ikp-swig-3.0.12/bin/swig (found version "3.0.12") 
CMake Warning at tf_shared_lib.cmake:136 (export):
  Cannot create package registry file:

    /homeless-shelter/.cmake/packages/Tensorflow/31aad99a164c4099f5d0af2d4ec07d6f

  No such file or directory

Call Stack (most recent call first):
  CMakeLists.txt:616 (include)


-- Configuring incomplete, errors occurred!
See also "/tmp/guix-build-tensorflow-1.15.0.drv-0/source/tensorflow/contrib/build/CMakeFiles/CMakeOutput.log".
See also "/tmp/guix-build-tensorflow-1.15.0.drv-0/source/tensorflow/contrib/build/CMakeFiles/CMakeError.log".
command "cmake" "../cmake" "-DCMAKE_BUILD_TYPE=Release" "-DCMAKE_INSTALL_PREFIX=/gnu/store/wm0iizlnsj55pdhjdm7k7pzz3bad84f0-tensorflow-1.15.0" "-DCMAKE_INSTALL_LIBDIR=lib" "-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=TRUE" "-DCMAKE_INSTALL_RPATH=/gnu/store/wm0iizlnsj55pdhjdm7k7pzz3bad84f0-tensorflow-1.15.0/lib" "-DCMAKE_VERBOSE_MAKEFILE=ON" "-Dprotobuf_STATIC_LIBRARIES=/gnu/store/p77n8kpsl50qlrz5fk0mc9kvfkinh9dq-protobuf-3.6.1/lib/libprotobuf.so" "-DPROTOBUF_PROTOC_EXECUTABLE=/gnu/store/p77n8kpsl50qlrz5fk0mc9kvfkinh9dq-protobuf-3.6.1/bin/protoc" "-Dsnappy_STATIC_LIBRARIES=/gnu/store/3xrpbdhhb8nk9p9jqr19ljlyhxnxk18n-snappy-1.1.8/lib/libsnappy.so" "-Dsnappy_INCLUDE_DIR=/gnu/store/3xrpbdhhb8nk9p9jqr19ljlyhxnxk18n-snappy-1.1.8" "-Djsoncpp_STATIC_LIBRARIES=/gnu/store/3vacq5lrnri4g8a5498qxkgxv4z8jyv8-jsoncpp-1.7.3/lib/libjsoncpp.so" "-Djsoncpp_INCLUDE_DIR=/gnu/store/3vacq5lrnri4g8a5498qxkgxv4z8jyv8-jsoncpp-1.7.3" "-Dsqlite_STATIC_LIBRARIES=/gnu/store/i6l1579g80387rda658jy9cfqq82643d-sqlite-3.28.0/lib/libsqlite.a" "-DABSEIL_CPP_LIBRARIES=/gnu/store/j36v4bb2vzy02xfyxp6ryfm4aj0yiaxn-abseil-cpp-20200225/lib/" "-DABSEIL_CPP_LIBRARIES_DIR_HINTS:STRING=/gnu/store/j36v4bb2vzy02xfyxp6ryfm4aj0yiaxn-abseil-cpp-20200225/lib/" "-Dsystemlib_ALL=ON" "-Dtensorflow_ENABLE_POSITION_INDEPENDENT_CODE=ON" "-Dtensorflow_BUILD_SHARED_LIB=ON" "-Dtensorflow_OPTIMIZE_FOR_NATIVE_ARCH=OFF" "-Dtensorflow_ENABLE_SSL_SUPPORT=OFF" "-Dtensorflow_BUILD_CONTRIB_KERNELS=OFF" failed with status 1
note: keeping build directory `/tmp/guix-build-tensorflow-1.15.0.drv-8'
builder for `/gnu/store/z9f8pcng2gsp7g9f93jd40c7g53wyc4s-tensorflow-1.15.0.drv' failed with exit code 1
build of /gnu/store/z9f8pcng2gsp7g9f93jd40c7g53wyc4s-tensorflow-1.15.0.drv failed
View build log at '/var/log/guix/drvs/z9/f8pcng2gsp7g9f93jd40c7g53wyc4s-tensorflow-1.15.0.drv.bz2'.
guix build: error: build of `/gnu/store/z9f8pcng2gsp7g9f93jd40c7g53wyc4s-tensorflow-1.15.0.drv' failed
--8<---------------cut here---------------end--------------->8---

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Request: Bazel build system (required for Tensorflow update)

[Roel Janssen]

On Wed, 2020-03-04 at 15:01 +0100, Pierre Neidhardt wrote:
> I've actually managed to fix the abseil issue.
> 
> Now I'm stuck at more broken CMake stuff.
> Duh, this will take a while...
> 
> --8<---------------cut here---------------start------------->8---
> -- Could NOT find c-ares (missing: c-ares_DIR)
> -- Found PythonInterp: /gnu/store/l8nphg0idd8pfddyad8f92lx8d1hc053-python-
> wrapper-3.7.4/bin/python (found version "3.7.4") 
> -- Found PythonLibs: /gnu/store/78w7y0lxar70j512iqw8x3nimzj10yga-python-
> 3.7.4/lib/libpython3.7m.so (found version "3.7.4") 
> CMake Error at tf_python.cmake:132 (message):
>   Python proto directory not found: tensorflow/contrib/tpu/profiler
> Call Stack (most recent call first):
>   CMakeLists.txt:613 (include)
> 
> 
> CMake Error at tf_python.cmake:217 (message):
>   Python module not found: tensorflow/contrib/rnn/kernels
> Call Stack (most recent call first):
>   CMakeLists.txt:613 (include)
> 
> 
> CMake Error at tf_python.cmake:217 (message):
>   Python module not found: tensorflow/contrib/rnn/ops
> Call Stack (most recent call first):
>   CMakeLists.txt:613 (include)
> 
> 
> CMake Error at tf_python.cmake:217 (message):
>   Python module not found: tensorflow/contrib/tpu/profiler
> Call Stack (most recent call first):
>   CMakeLists.txt:613 (include)
> 
> 
> -- Found SWIG: /gnu/store/1jamhp01xc911m68j8ndiwlcc55q8ikp-swig-
> 3.0.12/bin/swig (found version "3.0.12") 
> CMake Warning at tf_shared_lib.cmake:136 (export):
>   Cannot create package registry file:
> 
>     /homeless-
> shelter/.cmake/packages/Tensorflow/31aad99a164c4099f5d0af2d4ec07d6f
> 
>   No such file or directory
> 
> Call Stack (most recent call first):
>   CMakeLists.txt:616 (include)
> 
> 
> -- Configuring incomplete, errors occurred!
> See also "/tmp/guix-build-tensorflow-1.15.0.drv-
> 0/source/tensorflow/contrib/build/CMakeFiles/CMakeOutput.log".
> See also "/tmp/guix-build-tensorflow-1.15.0.drv-
> 0/source/tensorflow/contrib/build/CMakeFiles/CMakeError.log".
> command "cmake" "../cmake" "-DCMAKE_BUILD_TYPE=Release" "-
> DCMAKE_INSTALL_PREFIX=/gnu/store/wm0iizlnsj55pdhjdm7k7pzz3bad84f0-tensorflow-
> 1.15.0" "-DCMAKE_INSTALL_LIBDIR=lib" "-
> DCMAKE_INSTALL_RPATH_USE_LINK_PATH=TRUE" "-
> DCMAKE_INSTALL_RPATH=/gnu/store/wm0iizlnsj55pdhjdm7k7pzz3bad84f0-tensorflow-
> 1.15.0/lib" "-DCMAKE_VERBOSE_MAKEFILE=ON" "-
> Dprotobuf_STATIC_LIBRARIES=/gnu/store/p77n8kpsl50qlrz5fk0mc9kvfkinh9dq-
> protobuf-3.6.1/lib/libprotobuf.so" "-
> DPROTOBUF_PROTOC_EXECUTABLE=/gnu/store/p77n8kpsl50qlrz5fk0mc9kvfkinh9dq-
> protobuf-3.6.1/bin/protoc" "-
> Dsnappy_STATIC_LIBRARIES=/gnu/store/3xrpbdhhb8nk9p9jqr19ljlyhxnxk18n-snappy-
> 1.1.8/lib/libsnappy.so" "-
> Dsnappy_INCLUDE_DIR=/gnu/store/3xrpbdhhb8nk9p9jqr19ljlyhxnxk18n-snappy-1.1.8"
> "-Djsoncpp_STATIC_LIBRARIES=/gnu/store/3vacq5lrnri4g8a5498qxkgxv4z8jyv8-
> jsoncpp-1.7.3/lib/libjsoncpp.so" "-
> Djsoncpp_INCLUDE_DIR=/gnu/store/3vacq5lrnri4g8a5498qxkgxv4z8jyv8-jsoncpp-
> 1.7.3" "-Dsqlite_STATIC_LIBRARIES=/gnu/store/i6l1579g80387rda658jy9cfqq82643d-
> sqlite-3.28.0/lib/libsqlite.a" "-
> DABSEIL_CPP_LIBRARIES=/gnu/store/j36v4bb2vzy02xfyxp6ryfm4aj0yiaxn-abseil-cpp-
> 20200225/lib/" "-
> DABSEIL_CPP_LIBRARIES_DIR_HINTS:STRING=/gnu/store/j36v4bb2vzy02xfyxp6ryfm4aj0y
> iaxn-abseil-cpp-20200225/lib/" "-Dsystemlib_ALL=ON" "-
> Dtensorflow_ENABLE_POSITION_INDEPENDENT_CODE=ON" "-
> Dtensorflow_BUILD_SHARED_LIB=ON" "-Dtensorflow_OPTIMIZE_FOR_NATIVE_ARCH=OFF"
> "-Dtensorflow_ENABLE_SSL_SUPPORT=OFF" "-Dtensorflow_BUILD_CONTRIB_KERNELS=OFF" 
> failed with status 1
> note: keeping build directory `/tmp/guix-build-tensorflow-1.15.0.drv-8'
> builder for `/gnu/store/z9f8pcng2gsp7g9f93jd40c7g53wyc4s-tensorflow-
> 1.15.0.drv' failed with exit code 1
> build of /gnu/store/z9f8pcng2gsp7g9f93jd40c7g53wyc4s-tensorflow-1.15.0.drv
> failed
> View build log at '/var/log/guix/drvs/z9/f8pcng2gsp7g9f93jd40c7g53wyc4s-
> tensorflow-1.15.0.drv.bz2'.
> guix build: error: build of `/gnu/store/z9f8pcng2gsp7g9f93jd40c7g53wyc4s-
> tensorflow-1.15.0.drv' failed
> --8<---------------cut here---------------end--------------->8---
> 

Alright!  I've sent patches for abseil-cpp to the mailing list, but I haven't
been able to get the googletest test suite to work.  Have you figured that out?

Perhaps we can share the work on Tensorflow 1.15.2.  Would you mind sharing the
patch you've got so far?

Kind regards,
Roel Janssen

Re: Request: Bazel build system (required for Tensorflow update)

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 546 bytes --]

Roel Janssen <roel@gnu.org> writes:

> Alright!  I've sent patches for abseil-cpp to the mailing list, but I haven't
> been able to get the googletest test suite to work.  Have you figured that out?

I've reused your patch and fixed the tests.  It's a weird thing that
we've got with the googletest package, I'm not sure why this is happening.

> Perhaps we can share the work on Tensorflow 1.15.2.  Would you mind sharing the
> patch you've got so far?

Nothing significant, really :(

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Request: Bazel build system (required for Tensorflow update)

[Roel Janssen]

On Wed, 2020-03-04 at 16:10 +0100, Pierre Neidhardt wrote:
> Roel Janssen <roel@gnu.org> writes:
> 
> > Alright!  I've sent patches for abseil-cpp to the mailing list, but I
> > haven't
> > been able to get the googletest test suite to work.  Have you figured that
> > out?
> 
> I've reused your patch and fixed the tests.  It's a weird thing that
> we've got with the googletest package, I'm not sure why this is happening.

Well, thanks for fixing that!  It looks better than I thought it would like. :)

> > Perhaps we can share the work on Tensorflow 1.15.2.  Would you mind sharing
> > the
> > patch you've got so far?
> 
> Nothing significant, really :(
> 

Alright.

Kind regards,
Roel Janssen