From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id OJmaNLIf32AvOQEAgWs5BA (envelope-from ) for ; Fri, 02 Jul 2021 16:16:18 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id 6OEyMLIf32ARFgAAbx9fmQ (envelope-from ) for ; Fri, 02 Jul 2021 14:16:18 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 213A723349 for ; Fri, 2 Jul 2021 16:16:17 +0200 (CEST) Received: from localhost ([::1]:42252 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lzJxk-0006Yp-33 for larch@yhetil.org; Fri, 02 Jul 2021 10:16:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35892) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lzGvJ-0005LR-T0 for help-guix@gnu.org; Fri, 02 Jul 2021 07:01:30 -0400 Received: from mail-qk1-x732.google.com ([2607:f8b0:4864:20::732]:44836) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lzGvG-0005zq-BH for help-guix@gnu.org; Fri, 02 Jul 2021 07:01:29 -0400 Received: by mail-qk1-x732.google.com with SMTP id a4so1030432qkn.11 for ; Fri, 02 Jul 2021 04:01:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=tSUAWo+o0Xtf00o1ARbV1uTpUn7cu6HpthpS+KmAzFw=; b=gASBvtpIPXuhacqz4aNnIaWwMZKSQ8PzTYWWf99wufGZN8Bx9GLNJqyY1VTxQ043J9 JF4DxWxuhBp4EhTFnWPhEylLbfIYmggM6P6geE8cCvYJfzrejZb9MFVOvx84UAqRsyEg 8SaxhpAwrA0INLNTWeEbZPp1ARsF25Xp8RNuyU+NJhuQVyVqludJlYuAwRKE61jpZ06o f29Hy4MCMSoYbqnpIvKPMbCdwZaMipmBX4GVG5OatKEldObQQVtRhwVWPJMwHr1+H5/v Wxhtt+MZv3fbyK8wlS/WQ04npP82Bfq+/GhdNrV63Asy3Q1mWGhb1CZSla1/Kog/8hEa t2ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=tSUAWo+o0Xtf00o1ARbV1uTpUn7cu6HpthpS+KmAzFw=; b=aHQc6eKOq/P3D3KtW5tuM6z5QpkS+FGHay+hhcDP6ztiELJ2rnNOwD8aj9YJGIl2ye q/ULN+UPNxsvUfaebK27mcWcPccbRSlDcUuJBBiTTgU4rLLMQwiOONLOXaM2M7ZCxudp 6O1wno+yQmPRiDhF5YDnsZkHVlMOreWVBXu40kCXwelljVRE0B8mWvVoidMRSBjECXxG YL3fYFqRfWBtrFoiztAzpaQyha0WvNMeA8HKN9Vlna47yvVr7zKT9pajr+xvFYGPA+K9 yKo6zVMl5NPhWHR54aK6C3slxNRaM1mBWEE29wKOQRt0iAa+PpGYihtzjQFLdx1o5gQK jBaw== X-Gm-Message-State: AOAM531fi2Sdm8kVuao4kPCdtPca+FqI44KxbwfF+LmQHL+pPiLCpplk gdYuRR4EpTK7/CrYK+AeYOHfvHQySa7Dq77wDxmHHQ5dKq4= X-Google-Smtp-Source: ABdhPJzhwsAGOqwMiIRpA9FbARd39Z6ye46V6KURC/dSv+pNCpPENFKKuIBGn9TCMXU70gk2jQwyVS3Cxoipz1lkxxY= X-Received: by 2002:a37:ef0d:: with SMTP id j13mr4926742qkk.126.1625223683102; Fri, 02 Jul 2021 04:01:23 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: zimoun Date: Fri, 2 Jul 2021 13:01:11 +0200 Message-ID: Subject: Re: python-pyarrow broken for parquet? To: Phil Beadling Content-Type: text/plain; charset="UTF-8" Received-SPF: pass client-ip=2607:f8b0:4864:20::732; envelope-from=zimon.toutoune@gmail.com; helo=mail-qk1-x732.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Fri, 02 Jul 2021 10:15:54 -0400 X-BeenThere: help-guix@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: help-guix Errors-To: help-guix-bounces+larch=yhetil.org@gnu.org Sender: "Help-Guix" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1625235378; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=tSUAWo+o0Xtf00o1ARbV1uTpUn7cu6HpthpS+KmAzFw=; b=UZ11edlPAnHXeNoWgDTYICONrbpe+IA4QQG0ubUoM6TTA0nNzuRvK4JGoglVbknVIH0v7y fLQnTB7lJGuU9vs9wJdbJ91ZCzUkd29gsOOS6bjhFxcbDct+DN7b9pBD8MG1ZW7V70V8Qb kGT8RbeW1pfHKCCO/swa2WTGvU8mnmAUnSrfFGYtk7F/I0uxnevqlOUfxFaD5jopww4lgc 1wnF10SQZLivJ+z53x65fMjNYVs5xl72qGdAsLfUQp707GpSb9nDn4KuAZo9bvkQ1o05tK Q+RoXtDOk6tL+9xQnGSYJHN2wQL7rq17WdAKaQEWMrVfJEoWSbxeMCQ0oWesdw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1625235378; a=rsa-sha256; cv=none; b=KBVWI1SpoHb1gEfEL2mUzu7ySbKIuSM2k/MZO6ktU7sf9mWUV9rfH7IRjFNxSYkIPc2HwB tr+obAEJYEtWKyc8O3XnurD1ULi4yw6mkB6iQMcqg4F6gj17v808GdYULXr4mdiicV5tTz Z1MrrGLTmqKFeJr01NX9EkMWLLGvc+XFuU/R+YN8rFxXwsvHXQhMpLe8TQy35CY8eFVSCy Ou295bXC1bzOSrgCLeOczZcJx+wE5xD3n2ZP+DwOHvMvPbT0IdVThrxlki6VNeQ4nqRiB1 VfhIUsN9Z7zBZFDMrTICNCpkPA7IJQ6owvz//KR1NUstkGm8L+0BIqxeicqjqg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=gASBvtpI; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of help-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=help-guix-bounces@gnu.org X-Migadu-Spam-Score: -2.11 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=gASBvtpI; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of help-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=help-guix-bounces@gnu.org X-Migadu-Queue-Id: 213A723349 X-Spam-Score: -2.11 X-Migadu-Scanner: scn1.migadu.com X-TUID: hvCRTz7cupmr Hi, On Wed, 30 Jun 2021 at 21:10, Phil Beadling wrote: > When trying to use the parquet module in pyarrow - it's not finding the > internal module. This is true for latest 4.0.1 and also the previous > version (shown below). > > This is despite the fact that in the underlying apache-arrow package > parquet support is explicitly turned on as far as I can see: > https://github.com/guix-mirror/guix/blob/5ed105a8bb1a812975496dc3a091596355a0234c/gnu/packages/databases.scm#L3687 > > #:configure-flags > (list "-DARROW_PYTHON=ON" > "-DARROW_GLOG=ON" > ;; Parquet options > "-DARROW_PARQUET=ON" > "-DPARQUET_BUILD_EXECUTABLES=ON"..... The package 'apache-arrow' is built with the Parquet support but not the package 'python-pyarrow'. To add the support of Parquet to the Python binding, the environment variable PYTHON_WITH_PARQUET should be set to 1, IIRC the doc. Tweaking the 'python-pyarrow' with: --8<---------------cut here---------------start------------->8--- (add-after 'unpack 'set-env (lambda _ (setenv "PYARROW_WITH_PARQUET" "1") #t))))) --8<---------------cut here---------------end--------------->8--- then running "./pre-inst-env guix build python-pyarrow --no-grafts" I get: --8<---------------cut here---------------start------------->8--- -- Found Arrow: /gnu/store/bx4lprdcq5zw372f2rcc27c412chximr-apache-arrow-4.0.1-include/share/include (found version "4.0.1") -- Arrow version: 4.0.1 (CMake package configuration: Arrow) -- Arrow SO and ABI version: 400 -- Arrow full SO version: 400.1.0 -- Found the Arrow core shared library: /gnu/store/cagr2n03155mdx1mim8cxva1v22qhh0y-apache-arrow-4.0.1-lib/lib/libarrow.so -- Found the Arrow core import library: -- Found the Arrow core static library: -- Could NOT find ArrowPython (missing: ArrowPython_DIR) -- Checking for module 'arrow-python' -- Found arrow-python, version 4.0.1 -- Found ArrowPython: /gnu/store/bx4lprdcq5zw372f2rcc27c412chximr-apache-arrow-4.0.1-include/share/include (found version "4.0.1") -- Found the Arrow Python by pkg-config: arrow-python -- Found the Arrow Python shared library: /gnu/store/cagr2n03155mdx1mim8cxva1v22qhh0y-apache-arrow-4.0.1-lib/lib/libarrow_python.so -- Found the Arrow Python import library: -- Found the Arrow Python static library: ARROW_PYTHON_static_lib-NOTFOUND -- Could NOT find Parquet (missing: Parquet_DIR) -- Checking for module 'parquet' -- Found parquet, version 4.0.1 -- Found Parquet: /gnu/store/cagr2n03155mdx1mim8cxva1v22qhh0y-apache-arrow-4.0.1-lib//gnu/store/bx4lprdcq5zw372f2rcc27c412chximr-apache-arrow-4.0.1-include/share/include (found version "4.0.1") -- Parquet version: 4.0.1 (pkg-config: parquet) -- Found the Parquet shared library: /gnu/store/cagr2n03155mdx1mim8cxva1v22qhh0y-apache-arrow-4.0.1-lib/lib/libparquet.so -- Found the Parquet import library: -- Found the Parquet static library: PARQUET_static_lib-NOTFOUND -- Configuring done CMake Error in CMakeLists.txt: Imported target "parquet_shared" includes non-existent path "/gnu/store/cagr2n03155mdx1mim8cxva1v22qhh0y-apache-arrow-4.0.1-lib//gnu/store/bx4lprdcq5zw372f2rcc27c412chximr-apache-arrow-4.0.1-include/share/include" in its INTERFACE_INCLUDE_DIRECTORIES. Possible reasons include: * The path was deleted, renamed, or moved to another location. * An install or uninstall procedure did not complete successfully. * The installation package was faulty and references files it does not provide. CMake Error in CMakeLists.txt: Imported target "parquet_shared" includes non-existent path "/gnu/store/cagr2n03155mdx1mim8cxva1v22qhh0y-apache-arrow-4.0.1-lib//gnu/store/bx4lprdcq5zw372f2rcc27c412chximr-apache-arrow-4.0.1-include/share/include" in its INTERFACE_INCLUDE_DIRECTORIES. Possible reasons include: * The path was deleted, renamed, or moved to another location. * An install or uninstall procedure did not complete successfully. * The installation package was faulty and references files it does not provide. --8<---------------cut here---------------end--------------->8--- Without this environment variable PYTHON_WITH_PYARROW set to 1, the configuration is not looking after Parquet, for instance: --8<---------------cut here---------------start------------->8--- -- Found Arrow: /gnu/store/bx4lprdcq5zw372f2rcc27c412chximr-apache-arrow-4.0.1-include/share/include (found version "4.0.1") -- Arrow version: 4.0.1 (CMake package configuration: Arrow) -- Arrow SO and ABI version: 400 -- Arrow full SO version: 400.1.0 -- Found the Arrow core shared library: /gnu/store/cagr2n03155mdx1mim8cxva1v22qhh0y-apache-arrow-4.0.1-lib/lib/libarrow.so -- Found the Arrow core import library: -- Found the Arrow core static library: -- Could NOT find ArrowPython (missing: ArrowPython_DIR) -- Checking for module 'arrow-python' -- Found arrow-python, version 4.0.1 -- Found ArrowPython: /gnu/store/bx4lprdcq5zw372f2rcc27c412chximr-apache-arrow-4.0.1-include/share/include (found version "4.0.1") -- Found the Arrow Python by pkg-config: arrow-python -- Found the Arrow Python shared library: /gnu/store/cagr2n03155mdx1mim8cxva1v22qhh0y-apache-arrow-4.0.1-lib/lib/libarrow_python.so -- Found the Arrow Python import library: -- Found the Arrow Python static library: ARROW_PYTHON_static_lib-NOTFOUND -- Configuring done -- Generating done -- Build files have been written to: /tmp/guix-build-python-pyarrow-4.0.1.drv-0/source/python/build/temp.linux-x86_64-3.8 -- Finished cmake for pyarrow -- Running cmake --build for pyarrow cmake --build . --config release -- make[1]: Entering directory '/tmp/guix-build-python-pyarrow-4.0.1.drv-0/source/python/build/temp.linux-x86_64-3.8' --8<---------------cut here---------------end--------------->8--- Well, setting the environment variable is not enough. :-) Do you want to give a try for fixing it? All the best, simon