From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id uJ7pHhr44mBHXgAAgWs5BA (envelope-from ) for ; Mon, 05 Jul 2021 14:16:26 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id QLeYGhr44mD9dQAAB5/wlQ (envelope-from ) for ; Mon, 05 Jul 2021 12:16:26 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id B5B74D813 for ; Mon, 5 Jul 2021 14:16:25 +0200 (CEST) Received: from localhost ([::1]:58192 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m0NWS-0002mr-QV for larch@yhetil.org; Mon, 05 Jul 2021 08:16:24 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38318) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m0NW3-0002mi-Is for help-guix@gnu.org; Mon, 05 Jul 2021 08:15:59 -0400 Received: from mail-il1-x129.google.com ([2607:f8b0:4864:20::129]:42812) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1m0NW1-0000cs-PK for help-guix@gnu.org; Mon, 05 Jul 2021 08:15:59 -0400 Received: by mail-il1-x129.google.com with SMTP id h3so17151963ilc.9 for ; Mon, 05 Jul 2021 05:15:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=beadling-co-uk.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=tEAx8AGDXxA7fR81PH9/gcYFuWT+foZEOpBXCWvccHo=; b=AQ+lyWAnwGcbPDttftorg4wgEOmBFPvS9+Y9h94+0GgmmySjln0JHaQdx0LITHMz3y eFPmWDm6DWFoGi+4wsenkKmhncyB6u7kkQQBhqaSAppl95cJot6oI/dKOIUwVYqHkrW1 Ll1U8KbK30BX0CTwAiTyUboUe9PODGUhtAU4O7SCkTO3woDiNcdxCX9h/jNIhuHb6Bcg xD2SQ7y2BIasPdaEuEd/d71RpI8evSWcV14NA99uWmIbGPDcsRI0bhOH3VyvUZ3eSE6f AEEJsgcd/4y48jb2nBFYmhzC7bxblEMEKSzI6kvB1IXfRKcI55zsV2DVMYQmNVBNi9Ue uRnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=tEAx8AGDXxA7fR81PH9/gcYFuWT+foZEOpBXCWvccHo=; b=Ry0+nKM0yM6WJM20puyj6yyb4M0ceP+uTPINv4R/7arn5sc9QnAWSgzYlBMY0mbbDV Yfu/tZeY+CWT9warV4sOM98v1SYlxj3Jcpg2AAZ1iyGmloM0h7aBWdNcKpy+sG7uBzzs JAhT8cOHFSc4QBA1BaHImNlweOTqwYFL7iKXdAqwbeSlJE0VvXoZhGac5ormXHKL1DSP i575RpGr2BO/a6SWnFZwce33u5fdB9PDGfvzq+OIWgaUesYpbpQUBMeX8f1jGx1RAxmK JXRvrqG1zB8Rtw3rjnCnykY50TtcvDDZTOkBx4MD8oBisV56TWUrb9QQ0F4rRzQ3lEBF Qtqg== X-Gm-Message-State: AOAM533gTtlChlAzyyI+vecYW9tuik5SUawMcV9OyWmaZwBlmBOCfwZ0 OJJMqvPSEgbJiH/WP09d0XZmVKobcutyWqhudMdUFg== X-Google-Smtp-Source: ABdhPJyhUCXGLgmyR7Yrs34oWzY1n+ju+cd7M1fy+T2vcRzFfJPn9EX1IR+iYa87MJ5QQ6iSOoYYpkQSuNbhrjPEAHU= X-Received: by 2002:a92:b308:: with SMTP id p8mr7662033ilh.296.1625487356392; Mon, 05 Jul 2021 05:15:56 -0700 (PDT) MIME-Version: 1.0 References: <072ce53e-5d7c-478c-87a6-53e4e3ad6bb7.maildroid@localhost> In-Reply-To: From: Phil Beadling Date: Mon, 5 Jul 2021 13:15:45 +0100 Message-ID: Subject: Re: python-pyarrow broken for parquet? To: zimoun Received-SPF: none client-ip=2607:f8b0:4864:20::129; envelope-from=phil@beadling.co.uk; helo=mail-il1-x129.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: help-guix@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: help-guix Errors-To: help-guix-bounces+larch=yhetil.org@gnu.org Sender: "Help-Guix" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1625487386; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=IxtlHNGkANjAeSxwaWo3EuZNxgxKK71sbE5DBrbNDwM=; b=uTinPpB8pRI6qbHapj5nfr4YNOZK5P6VnMmCpp9Q9M0Qhl6IlVYIzqeIqCNNtqGdS3VFd+ x/gk0EibS5FQ1h4nP45UWounVamxZFgBovQALjgO5OUygLrPTvbL5/FB+nyI305vkqZq19 lCgabmvLOCJWT1nMoCKiTS5nw0KGwPDqS99JM7W9SwPu/C0qg80mUYmxVYxKGcVd4zHTuZ SRyAQWJdD6Hp7uWGXpzBSW/0YadBeEaWm9UZ1oiXzUu9YywTREXOP0CF8srqI7+K9ydoI3 YeUQDVndlJoEH/rvXyyLvalXhUlYzgu7Xvl2kHKsDJzz3OloVLbuvgJu2f+mBQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1625487386; a=rsa-sha256; cv=none; b=ux15HfDY6ILK22LS3gW1KJi/m6Hj2OFMqQtSDX9vYtENeC7ptr9aTndxpt1/xaLHqpWtWP zOylW88MOgLpmh8mmoMifYdB7+3QeyIfXoyHPq0Xex/6AZ0SGfIwN17UamsEHt4YobTKO6 riK7NeetVFf2ydK9pFAW/iT45lFCtnxjXu4+QOIPvYHKAUqad802UV2ORllVdF44Oa2c9v 4km4x75GU/0xQzLW7qLbbuFAEk6HE8Udl2/HvCayenW71MQc+CXnkhIhjTeFx4q6VRwv6p ehwN975dAuaUapftgm+W1MLakdkqAoiXq1M4AIlTWz+9tFtDHIUBaL3qcSMhIQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=beadling-co-uk.20150623.gappssmtp.com header.s=20150623 header.b=AQ+lyWAn; dmarc=none; spf=pass (aspmx1.migadu.com: domain of help-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=help-guix-bounces@gnu.org X-Migadu-Spam-Score: -0.41 Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=beadling-co-uk.20150623.gappssmtp.com header.s=20150623 header.b=AQ+lyWAn; dmarc=none; spf=pass (aspmx1.migadu.com: domain of help-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=help-guix-bounces@gnu.org X-Migadu-Queue-Id: B5B74D813 X-Spam-Score: -0.41 X-Migadu-Scanner: scn0.migadu.com X-TUID: nKRlclCVyJQL Apologies one tiny correction - the version of python-pandas probably doesn't need changing - in the package I update to a more recent custom build of pandas, I'm using - but it's probably not necessary: ("python-pandas" ,python-pandas-simm) On Mon, 5 Jul 2021 at 13:13, Phil Beadling wrote: > As promised - this works for me but the patching of the make files, in > particular the 2 sed commands is very brittle to any changes in the > underlying project. I'm not sure it should go into Guix proper as-is, but > if people think it's useful I'm happy to submit the patch. > > I may try to improve on this when I have moment, but if someone else wants > to run with it - it's a good starting point at least. > > The problem is that the generation of the PARQUET_INCLUDE_DIR nad > PARQUET_LIB_DIR end up concatentating both the include and lib dirs > together: > > For example debugging the cmake file the value > "PARQUET_INCLUDE_DIR/parquet" becomes: > "*/gnu/store/ywklhws3ccb457gsb605z95azbfpsbyl-apache-arrow-3.0.0-lib/*/gnu/store/zzzb4ymfj3igynsflxwxsn58kvnpa6qb-apache-arrow-3.0.0-include/share/include/parquet" > > > The *lib *directory shouldn't be there at all. > > --8<---------------cut here---------------start------------->8--- > (define-public python-pyarrow-parquet > (package/inherit python-pyarrow > (arguments > (substitute-keyword-arguments (package-arguments > python-pyarrow) > ((#:phases phases) > `(modify-phases ,phases > (add-before 'install 'patch-cmake-variables > (lambda* (#:key inputs #:allow-other-keys) > ;; Replace cmake locations with hardcoded > guix links for the underlying C++ lib - this is a pretty awful hack > (invoke "sed" "-i" (string-append > "1s#^#set(PARQUET_INCLUDE_DIR \"" (assoc-ref inputs "apache-arrow:include") > "/share/include\ > \")\\n#") "cmake_modules/FindParquet.cmake") > (invoke "sed" "-i" (string-append > "116s#^#set(PARQUET_LIB_DIR \"" (assoc-ref inputs "apache-arrow:lib") > "/lib\")\\n#") "cmake\ > _modules/FindParquet.cmake"))) > (add-before 'install 'patch-parquet-library > (lambda _ > ;; Another nasty hack - there must be a > better way to change this? > (substitute* "CMakeLists.txt" > (("parquet_shared") "parquet")))) > (add-before 'install 'set-PYARROW_WITH_PARQUET > (lambda _ > (setenv "PYARROW_WITH_PARQUET" "1") > ;;(setenv "VERBOSE" "1") ;; useful debug for > cmake > #t)))))) > ;; we need includes from apache as well as libs for > parquet > (propagated-inputs > `(("python-pandas" ,python-pandas-simm) > ("apache-arrow:lib" ,apache-arrow "lib") > ("apache-arrow:include" ,apache-arrow "include") > ,@(fold alist-delete (package-propagated-inputs > python-pyarrow) > '("python-pandas" "apache-arrow")))))) > --8<---------------cut here---------------end--------------->8--- > > On Fri, 2 Jul 2021 at 16:34, wrote: > >> Thanks Simon. >> >> Yep I got this far too - and I have a candidate fix for building >> parquet. But it's tremendously hacky (sed'ing hardcoded variables into the >> cmake files to trample the derived settings in several places). It seems >> to work but needs finessing. I'll post here shortly, but not sure it's >> stable enough to be updated in Guix proper. We can debate that when >> everyone sees my horrendous fix. >> >> >>