From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id SHioCp734mBAXAAAgWs5BA (envelope-from ) for ; Mon, 05 Jul 2021 14:14:22 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id ADBRBp734mAmTgAA1q6Kng (envelope-from ) for ; Mon, 05 Jul 2021 12:14:22 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 7A149178B7 for ; Mon, 5 Jul 2021 14:14:21 +0200 (CEST) Received: from localhost ([::1]:57558 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m0NUS-0001qH-Ag for larch@yhetil.org; Mon, 05 Jul 2021 08:14:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38056) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m0NUE-0001i2-Aa for help-guix@gnu.org; Mon, 05 Jul 2021 08:14:06 -0400 Received: from mail-io1-xd2f.google.com ([2607:f8b0:4864:20::d2f]:40947) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1m0NUB-0007y3-Rx for help-guix@gnu.org; Mon, 05 Jul 2021 08:14:06 -0400 Received: by mail-io1-xd2f.google.com with SMTP id l5so20707820iok.7 for ; Mon, 05 Jul 2021 05:14:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=beadling-co-uk.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=nZ36Dq2LX9eKUdsyQ0EWFpW41JxBhJM9m81uXgafu6w=; b=lBMXY1H4zB/vc9eYyaNtDfMSnv6YBSlwadlGOVAzyE2vRGNFjt01kxu80FNlDrndUq KKEw1XCnHrYeCzwiO4kiiL9GWLzDmB6g+3rLHF4G2Y0y3SjV/eMJaKK5icvCB9feq1wt JT+cB72k//O34VV3JRLXDuPUdL43uF2hrmY7pFpD0bG7lOzpwN5hONZZYaNnahYwGLAf OfQ+4CnbjjFZSA403eqrSUFgp+uemQba4BsOHRH9R60KsJ3tEXuDSsiWL9xZcKDt2DoJ MHkpq8T8EpP0Z2mynZAIwtVhruzRjjGiSTUc+ZaZquR/yJCqXMQ0xUdATLeX9B2opbM1 /1DQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=nZ36Dq2LX9eKUdsyQ0EWFpW41JxBhJM9m81uXgafu6w=; b=igqhUmkiKYW41dIIftJvXUx5KNoR4/nItLn5Slf7jLOvfWxoxTkBfZ+Ce3afNRFKOx 6QDNZIJnsGlPNjwRua+x/cTTox4TyNrFSxzOrnttiMMBe/0vWtL0N7/+6N7s4YYU9EGl 3OCKGPwMOC4B2PNVEdQkJpl/4nytO4zWvCzT4GgtW7YYf6B0kCUhg2I2056qieClMaiL zfUB/Bk5nlMD9Yln37ZpjKt/JaN+YGKFjme9hV4kWze0SQtynsXNxrKXt2BfDf+rNWv9 t+fbLBeFh/V1Y06FqILe+6ScGAGevhqfWs88nGit/l3a2Vb/ld3zvULRW4mIqwF4CPPR bE1w== X-Gm-Message-State: AOAM533X8vbExerNNbjSpKLbg1ZhhzUvxHNHy/ZJTY35bGMpdY1qtQ8F zXAzKy9UBBr43dauz9PJRlZwhzqnkNL1fX7VgNSsMBOjDTeQ1cZY0J4= X-Google-Smtp-Source: ABdhPJySAd9eRDQr6VGlojuZHM33n+rlSgCf8KHN8QayLD6qao9s4VctpuVBIyahRq42Dp2OAehnII2ze6So9ko/oN4= X-Received: by 2002:a05:6638:2612:: with SMTP id m18mr12367999jat.54.1625487242051; Mon, 05 Jul 2021 05:14:02 -0700 (PDT) MIME-Version: 1.0 References: <072ce53e-5d7c-478c-87a6-53e4e3ad6bb7.maildroid@localhost> In-Reply-To: <072ce53e-5d7c-478c-87a6-53e4e3ad6bb7.maildroid@localhost> From: Phil Beadling Date: Mon, 5 Jul 2021 13:13:51 +0100 Message-ID: Subject: Re: python-pyarrow broken for parquet? To: zimoun Received-SPF: none client-ip=2607:f8b0:4864:20::d2f; envelope-from=phil@beadling.co.uk; helo=mail-io1-xd2f.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: help-guix@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: help-guix Errors-To: help-guix-bounces+larch=yhetil.org@gnu.org Sender: "Help-Guix" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1625487261; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=vVJJYJgS639Naedas2xzwpSpQ20GtEf015pu50kVoC0=; b=f0d83euF7st0zc9asXkJjDFcJxW3yIHiwbdMCPWUNQjZVnvKVH14EykMmj7hNGMWuOl4YT qj5WNkJs3d261kVfMHnsBLEvlmqVbobNhM8v57xmOWQW2gY8mX97VTY/4wiix4AGqQOKlq s5hAK413Wrtzj5yAT72UlUBihXFqg3zIVshNCr1XzJA97csrvDrX5xE3CBroutxCx4opSL o/kKXw7NkoKbGjFrZeaGeh+PzDLErjtUZAOyjmYfuazcN8Pu7+eN4GeDVIIvU6pefHb9bE qXDp+KAnaqBkxb4lDLfxsLevc2hpMAvWfhrTOspmrD1ZkQiTmSmqjk94J0abUQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1625487261; a=rsa-sha256; cv=none; b=L9N4Q1v9Es56lKcZAZiBE7YawyPKsQDSLCFl58PwrmvQq0Y4zl8yus01gtw8H+8f4KXXhQ 2aDBEJZnXsChYllh9CkXmVNZtM+VLmxxGIN7X8qD2KrxZDxqO6RkaZaCvXCL3Xi0Vou/o3 heLHzqjczORYOwzUKYGP0C7endRjDruYt/iYWPUVcXFGhLtymuQm4vnxhZQWJOaoVQafZY sfS4POKHPN62NWZ4CLMIz96MqeNK2L+M0BW69yyctmcSxCJT0zhzq4v07+lDb2zH9QpLUq A35UChusktKMt2Fm3HR/hWBTvPDHtmrUvsr2rLvey0xRRVGPXIRqTO/2UZVqpw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=beadling-co-uk.20150623.gappssmtp.com header.s=20150623 header.b=lBMXY1H4; spf=pass (aspmx1.migadu.com: domain of help-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=help-guix-bounces@gnu.org X-Migadu-Spam-Score: -0.41 Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=beadling-co-uk.20150623.gappssmtp.com header.s=20150623 header.b=lBMXY1H4; dmarc=none; spf=pass (aspmx1.migadu.com: domain of help-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=help-guix-bounces@gnu.org X-Migadu-Queue-Id: 7A149178B7 X-Spam-Score: -0.41 X-Migadu-Scanner: scn0.migadu.com X-TUID: JWjknNRLvcKw As promised - this works for me but the patching of the make files, in particular the 2 sed commands is very brittle to any changes in the underlying project. I'm not sure it should go into Guix proper as-is, but if people think it's useful I'm happy to submit the patch. I may try to improve on this when I have moment, but if someone else wants to run with it - it's a good starting point at least. The problem is that the generation of the PARQUET_INCLUDE_DIR nad PARQUET_LIB_DIR end up concatentating both the include and lib dirs together: For example debugging the cmake file the value "PARQUET_INCLUDE_DIR/parquet" becomes: "*/gnu/store/ywklhws3ccb457gsb605z95azbfpsbyl-apache-arrow-3.0.0-lib/*/gnu/store/zzzb4ymfj3igynsflxwxsn58kvnpa6qb-apache-arrow-3.0.0-include/share/include/parquet" The *lib *directory shouldn't be there at all. --8<---------------cut here---------------start------------->8--- (define-public python-pyarrow-parquet (package/inherit python-pyarrow (arguments (substitute-keyword-arguments (package-arguments python-pyarrow) ((#:phases phases) `(modify-phases ,phases (add-before 'install 'patch-cmake-variables (lambda* (#:key inputs #:allow-other-keys) ;; Replace cmake locations with hardcoded guix links for the underlying C++ lib - this is a pretty awful hack (invoke "sed" "-i" (string-append "1s#^#set(PARQUET_INCLUDE_DIR \"" (assoc-ref inputs "apache-arrow:include") "/share/include\ \")\\n#") "cmake_modules/FindParquet.cmake") (invoke "sed" "-i" (string-append "116s#^#set(PARQUET_LIB_DIR \"" (assoc-ref inputs "apache-arrow:lib") "/lib\")\\n#") "cmake\ _modules/FindParquet.cmake"))) (add-before 'install 'patch-parquet-library (lambda _ ;; Another nasty hack - there must be a better way to change this? (substitute* "CMakeLists.txt" (("parquet_shared") "parquet")))) (add-before 'install 'set-PYARROW_WITH_PARQUET (lambda _ (setenv "PYARROW_WITH_PARQUET" "1") ;;(setenv "VERBOSE" "1") ;; useful debug for cmake #t)))))) ;; we need includes from apache as well as libs for parquet (propagated-inputs `(("python-pandas" ,python-pandas-simm) ("apache-arrow:lib" ,apache-arrow "lib") ("apache-arrow:include" ,apache-arrow "include") ,@(fold alist-delete (package-propagated-inputs python-pyarrow) '("python-pandas" "apache-arrow")))))) --8<---------------cut here---------------end--------------->8--- On Fri, 2 Jul 2021 at 16:34, wrote: > Thanks Simon. > > Yep I got this far too - and I have a candidate fix for building parquet. > But it's tremendously hacky (sed'ing hardcoded variables into the cmake > files to trample the derived settings in several places). It seems to work > but needs finessing. I'll post here shortly, but not sure it's stable > enough to be updated in Guix proper. We can debate that when everyone sees > my horrendous fix. > > >