From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:8:6d80::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id SJBJLtG6eWDnFgEAgWs5BA (envelope-from ) for ; Fri, 16 Apr 2021 18:26:57 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id MCwrKNG6eWAdQwAA1q6Kng (envelope-from ) for ; Fri, 16 Apr 2021 16:26:57 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id B4B5B25CB5 for ; Fri, 16 Apr 2021 18:26:56 +0200 (CEST) Received: from localhost ([::1]:34488 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lXRIz-0000Nz-3n for larch@yhetil.org; Fri, 16 Apr 2021 12:26:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37880) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lXOyu-0001vs-Vk for guix-devel@gnu.org; Fri, 16 Apr 2021 09:58:00 -0400 Received: from mail-io1-xd32.google.com ([2607:f8b0:4864:20::d32]:36536) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lXOys-0006Hl-SG for guix-devel@gnu.org; Fri, 16 Apr 2021 09:58:00 -0400 Received: by mail-io1-xd32.google.com with SMTP id d5so18320442iof.3 for ; Fri, 16 Apr 2021 06:57:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=bmM6VslSXzfPLtT81FCLGy5MUmxMRP831VcoTeB0olQ=; b=NmkdemFyuvB6C0wejI1ij/IvOwMadUNF2+435NaG6mVvKcsaFCF6w7bOZZ/aOoBYlh 41D+6TTsfqiq1NkKMDLNpauuTa390Xgx6tQox92vELWt86nhze0QE3Vxw4x+U878jJpM v96BrXBEevs3Kcohg3CYo8A13u9z7D2QOWrBDgQ2h9FMUFLGPwAIc9B5r1AjUNhoBENt ibh/WcUkTshSOso8O6WbHGbNYPwfdwQCTbUqX0RjWxASCqvnGPOGHlPzgiihx9oiNsQV 7WIv9+t3F4R54aDcvUkSvn2FfT2je6ryy2WrvE5LnvCsAIink+IhgWwtc4IgrQS5Vk98 VcWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=bmM6VslSXzfPLtT81FCLGy5MUmxMRP831VcoTeB0olQ=; b=Ec8ADhu665YmwiITZwRRDZpIFYbMKi1lme0Y20/zTe8AYc93n6CzgiGuJy/Kke048E Pw1Z4yYWUqfUtS0ATs5ICrTgzCTVcWt10vcA6a6jzU9+lA5N67TTj82JSVGf16fUVbSK hedsLJmMLEUdIx33aNBTdcOaz2F1QrXbZxOVnW1jz8fzrqxCRYQnO4VnntU89bMwhb70 frKywR4KTQ9aKu3D8CXVxiXBdJvc1+Pc3CU3N94uI8WtwypeB/GgAQ4mK09VdilLqXuE kIwPb5m63JWq9OrXSH0xIKUgueBwX+FAucsgjnk9kl1GetP2P3W3jpoPVfXMJn8Dge8A mlog== X-Gm-Message-State: AOAM530sKoZNtsAl5MPSNPYE4NzL5jyVQjEKOc08iLatTHJTC4jkHwlL TAnP74XAy0RyTqIKWJJ/jZASKELTEAbkEb2O3BxCs056 X-Google-Smtp-Source: ABdhPJzn1o3Ner1D0EcWYdO4qr1ZR4aUAHzyZvdA9e9pv2cuBMqhsjd2x/6CZQqQy0k1Op4YumSusovqr1LTkLf+67U= X-Received: by 2002:a05:6638:3013:: with SMTP id r19mr4222125jak.36.1618581477597; Fri, 16 Apr 2021 06:57:57 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Vladilen Kozin Date: Fri, 16 Apr 2021 14:57:46 +0100 Message-ID: Subject: Re: Help adding a python package: chia-blockchain To: guix-devel@gnu.org Content-Type: text/plain; charset="UTF-8" Received-SPF: pass client-ip=2607:f8b0:4864:20::d32; envelope-from=vladilen.kozin@gmail.com; helo=mail-io1-xd32.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Fri, 16 Apr 2021 12:26:11 -0400 X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1618590416; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=bmM6VslSXzfPLtT81FCLGy5MUmxMRP831VcoTeB0olQ=; b=WVGTksXC/SLJELF9aUjbafYlNoz3B8nk3T1zWrXQFq7s4A/lKby22JwmOGOpO6goesqrAO Hjk5QBSz8OBz3VPJ78sdLbtrvo5a4LLmkXQzyEgfh7JRZAttW6H+KY955XTt/j9wO100wa Rhbl6YFqwinAl7Zj2D7OuUot+qvZittDpJZqZzX6lqpuURkNgnpIwQT5Nc2rpnVwaBa6Yi NYRcB0y54egEWdDemXsjxpRYsXiQMeWSVr0i3c7Q+Fs41mNSvRUMod124EBQbD8Uzrtjmm lP2gziz6G+WJ5in7dTyG7uHrCmMZRVTgCSryC1X5QvKW+mu5fyfrMOPgUMfmew== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1618590416; a=rsa-sha256; cv=none; b=Ma7N2D/LCKWcQD8ASfUzgSffHTKCvxb120glC/2ieCnDjkMh9VsBWMtZ6NDxmZ8uASdL/d gj5OW0s5AFNUS8cERGXpnJM88Yy0TBQSNKdr/gjIis9TaZcNmUMQc3LjPwMRxJG7+wMRqO QbZOglylXje801R4OTtI52aXsjtMcIJAeDY57TBkgTppG2tW2Ok8FYAeqX6CoJW23/bwhA oO0MVz7NJULajkuTuH0Wgj7EP0BOmkEmu6XeFtb1AIRZH03teXzAXmQVXkMurcaZ9JS5JA aylXV3RIj65/F/oV2kp4ojXhEsmKcZopbvR6HtY1muV2KUw/syv/S0Ti10ZXZA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=NmkdemFy; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -3.14 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=NmkdemFy; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: B4B5B25CB5 X-Spam-Score: -3.14 X-Migadu-Scanner: scn0.migadu.com X-TUID: BsLxhHj+QG/Q Another "hilarious" consequence of attempting to side-step "the guix" way. If I just follow the `pip install` steps I end up with some C++ dependency (blspy I believe) complaining that it can't find libstdc++. Quick `locate` search unsurfaced it somewhere under /gnu/store, but of course binaries (like binary wheels) pre-built this way are completely oblivious to Linux machines (or I suppose Guix) that don't have the usual /lib or /lib64 or whatever. So yeah, looks like its guix way or no way. Like I mentioned earlier, could use some guidance in packaging this. On Fri, 16 Apr 2021 at 09:57, Vladilen Kozin wrote: > > Hi all. > > Could a kind soul handhold me while I try to port a python package to > guix please. I am not a python programmer, so not on first name basis > with python ecosystem and new to guix - double trouble. > > I've made some progress, but could use a bit of your help. Current > state of affairs is tracked in this repo: > https://github.com/vkz/ze-guix > > Repo and package I'm trying to import: > https://github.com/Chia-Network/chia-blockchain > > Both it and all of its dependencies are open source, most on pypi but > sadly some are only available there as binary wheels. Building from > source entirely maybe complicated since it very quickly escalates to > multiple build systems e.g. chiapos (proof of space lib) and chiavdf > are C++ and rely on cmake, clvm_rs (their homegrown lisp) is written > in Rust etc. > > Conceptually, installation is very simple and basically relies on > binary wheels and amounts to: > - checkout repo, > - init and update submodule(s) - really only mozilla-ca, > - create virtual env and activate, > - upgrade to the latest pip to use binary wheels, > - install deps and the package with pip > > You can see the steps I extracted from their install.sh here: > https://github.com/vkz/ze-guix/blob/master/packages/manual-install.log > > Now, guix obviously heads and shoulders above virtualenv or at least > it should be, however trying to build from source escalates quickly as > I've mentioned. So, first question is if I could define a package by > "replicating" the above essentially binary-only steps but replace > virtualenv with e.g. guix profile or smth. This should be possible to > do in code - as a proper package with some rudimentary build system > that amounts to building everything inside a separate profile or maybe > just pulling binary wheels into a directory in store and then adding > that to whatever Python or pip expect and use as their PATH > equivalent. IMO doing something trivial like this should be easy. To > be sure this approach has security and "freedom" implications, but > this is orthogonal to having something that works installed quickly. > Building from source is very often involved borderline crazy. Could > someone kindly teach me how to do this "binary" only packaging. > > Now, to building from source. > > First step I tried was to `guix import pypi -r`. This fails because > not all deps have source pushed to pypi and only offer binary wheels. > IMO `guix import` has no business failing like this or even at all for > whatever reason. It is best effort tool to begin with that should > result in a template that I can fill in. If there are errors - skip > the dependency, report it, but please continue. > > Since, recursive `guix import` failed me I had to manually track deps > and essentially call `guix import` for each one. Result you can see > here: > https://github.com/vkz/ze-guix/blob/master/packages/chia.scm > > This won't build for several reasons. Not all deps are there, some > probably shouldn't be like win32 something or other. Another reason, > I'd have to figure out how to build the inputs that require cmake and > rust and python (some all at once). And this is where I could really > use help. Examples as mentioned earlier are these dependencies: > https://github.com/Chia-Network/chiapos > https://github.com/Chia-Network/clvm_rs > maybe others - it is a fairly big project and I've not even touched > NPM/node that they rely on for their GUI client. That's another scary > can of worms I'm not prepared to peek inside at this time. > > Ok, so really two questions: > - how can I build a "binary" only python package that simply relies on > binary wheels e.g. from pypi, > - how to deal with multi-build systems e.g. python + cmake or python + > rust, maybe someone could think of a good educational (so, not too > convoluted) example of such package already in source tree. > > Latter would probably go much faster and teach better if done as > pair-programming exercise, but that maybe a lot to ask. If you're up > for it, I'm game though. > > Thank you > -- > Best regards > Vlad Kozin -- Best regards Vlad Kozin