Re: python-pyarrow broken for parquet?

[zimoun <zimon.toutoune@gmail.com>]

Hi,

On Wed, 30 Jun 2021 at 21:10, Phil Beadling <phil@beadling.co.uk> wrote:

> When trying to use the parquet module in pyarrow - it's not finding the
> internal module.  This is true for latest 4.0.1 and also the previous
> version (shown below).
>
> This is despite the fact that in the underlying apache-arrow package
> parquet support is explicitly turned on as far as I can see:
> https://github.com/guix-mirror/guix/blob/5ed105a8bb1a812975496dc3a091596355a0234c/gnu/packages/databases.scm#L3687
>
> #:configure-flags
> (list "-DARROW_PYTHON=ON"
> "-DARROW_GLOG=ON"
> ;; Parquet options
> "-DARROW_PARQUET=ON"
> "-DPARQUET_BUILD_EXECUTABLES=ON".....

The package 'apache-arrow' is built with the Parquet support but not
the package 'python-pyarrow'.  To add the support of Parquet to the
Python binding, the environment variable PYTHON_WITH_PARQUET should be
set to 1, IIRC the doc.

Tweaking the 'python-pyarrow' with:

--8<---------------cut here---------------start------------->8---
      (add-after 'unpack 'set-env
        (lambda _ (setenv "PYARROW_WITH_PARQUET" "1") #t)))))
--8<---------------cut here---------------end--------------->8---

then running "./pre-inst-env guix build python-pyarrow --no-grafts" I get:

--8<---------------cut here---------------start------------->8---
-- Found Arrow:
/gnu/store/bx4lprdcq5zw372f2rcc27c412chximr-apache-arrow-4.0.1-include/share/include
(found version "4.0.1")
-- Arrow version: 4.0.1 (CMake package configuration: Arrow)
-- Arrow SO and ABI version: 400
-- Arrow full SO version: 400.1.0
-- Found the Arrow core shared library:
/gnu/store/cagr2n03155mdx1mim8cxva1v22qhh0y-apache-arrow-4.0.1-lib/lib/libarrow.so
-- Found the Arrow core import library:
-- Found the Arrow core static library:
-- Could NOT find ArrowPython (missing: ArrowPython_DIR)
-- Checking for module 'arrow-python'
--   Found arrow-python, version 4.0.1
-- Found ArrowPython:
/gnu/store/bx4lprdcq5zw372f2rcc27c412chximr-apache-arrow-4.0.1-include/share/include
(found version "4.0.1")
-- Found the Arrow Python by pkg-config: arrow-python
-- Found the Arrow Python shared library:
/gnu/store/cagr2n03155mdx1mim8cxva1v22qhh0y-apache-arrow-4.0.1-lib/lib/libarrow_python.so
-- Found the Arrow Python import library:
-- Found the Arrow Python static library: ARROW_PYTHON_static_lib-NOTFOUND
-- Could NOT find Parquet (missing: Parquet_DIR)
-- Checking for module 'parquet'
--   Found parquet, version 4.0.1
-- Found Parquet:
/gnu/store/cagr2n03155mdx1mim8cxva1v22qhh0y-apache-arrow-4.0.1-lib//gnu/store/bx4lprdcq5zw372f2rcc27c412chximr-apache-arrow-4.0.1-include/share/include
(found version "4.0.1")
-- Parquet version: 4.0.1 (pkg-config: parquet)
-- Found the Parquet shared library:
/gnu/store/cagr2n03155mdx1mim8cxva1v22qhh0y-apache-arrow-4.0.1-lib/lib/libparquet.so
-- Found the Parquet import library:
-- Found the Parquet static library: PARQUET_static_lib-NOTFOUND
-- Configuring done
CMake Error in CMakeLists.txt:
  Imported target "parquet_shared" includes non-existent path

    "/gnu/store/cagr2n03155mdx1mim8cxva1v22qhh0y-apache-arrow-4.0.1-lib//gnu/store/bx4lprdcq5zw372f2rcc27c412chximr-apache-arrow-4.0.1-include/share/include"

  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:

  * The path was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and references files it does not
  provide.


CMake Error in CMakeLists.txt:
  Imported target "parquet_shared" includes non-existent path

    "/gnu/store/cagr2n03155mdx1mim8cxva1v22qhh0y-apache-arrow-4.0.1-lib//gnu/store/bx4lprdcq5zw372f2rcc27c412chximr-apache-arrow-4.0.1-include/share/include"

  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:

  * The path was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and references files it does not
  provide.
--8<---------------cut here---------------end--------------->8---

Without this environment variable PYTHON_WITH_PYARROW set to 1, the
configuration is not looking after Parquet, for instance:

--8<---------------cut here---------------start------------->8---
-- Found Arrow:
/gnu/store/bx4lprdcq5zw372f2rcc27c412chximr-apache-arrow-4.0.1-include/share/include
(found version "4.0.1")
-- Arrow version: 4.0.1 (CMake package configuration: Arrow)
-- Arrow SO and ABI version: 400
-- Arrow full SO version: 400.1.0
-- Found the Arrow core shared library:
/gnu/store/cagr2n03155mdx1mim8cxva1v22qhh0y-apache-arrow-4.0.1-lib/lib/libarrow.so
-- Found the Arrow core import library:
-- Found the Arrow core static library:
-- Could NOT find ArrowPython (missing: ArrowPython_DIR)
-- Checking for module 'arrow-python'
--   Found arrow-python, version 4.0.1
-- Found ArrowPython:
/gnu/store/bx4lprdcq5zw372f2rcc27c412chximr-apache-arrow-4.0.1-include/share/include
(found version "4.0.1")
-- Found the Arrow Python by pkg-config: arrow-python
-- Found the Arrow Python shared library:
/gnu/store/cagr2n03155mdx1mim8cxva1v22qhh0y-apache-arrow-4.0.1-lib/lib/libarrow_python.so
-- Found the Arrow Python import library:
-- Found the Arrow Python static library: ARROW_PYTHON_static_lib-NOTFOUND
-- Configuring done
-- Generating done
-- Build files have been written to:
/tmp/guix-build-python-pyarrow-4.0.1.drv-0/source/python/build/temp.linux-x86_64-3.8
-- Finished cmake for pyarrow
-- Running cmake --build for pyarrow
cmake --build . --config release --
make[1]: Entering directory
'/tmp/guix-build-python-pyarrow-4.0.1.drv-0/source/python/build/temp.linux-x86_64-3.8'
--8<---------------cut here---------------end--------------->8---

Well, setting the environment variable is not enough.  :-)  Do you
want to give a try for fixing it?


All the best,
simon