From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:bcc0::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id yP11FIE8emB+PwEAgWs5BA (envelope-from ) for ; Sat, 17 Apr 2021 03:40:17 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id GPs3D4E8emA7LQAAbx9fmQ (envelope-from ) for ; Sat, 17 Apr 2021 01:40:17 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 486EB235C4 for ; Sat, 17 Apr 2021 03:40:16 +0200 (CEST) Received: from localhost ([::1]:54082 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lXZwT-00071J-IF for larch@yhetil.org; Fri, 16 Apr 2021 21:40:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53114) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lXZdO-0007nq-0w for guix-devel@gnu.org; Fri, 16 Apr 2021 21:20:30 -0400 Received: from mailrelay.tugraz.at ([129.27.2.202]:21136) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lXZdK-0004wQ-8q for guix-devel@gnu.org; Fri, 16 Apr 2021 21:20:29 -0400 Received: from nijino.local (194-96-13-79.adsl.highway.telekom.at [194.96.13.79]) by mailrelay.tugraz.at (Postfix) with ESMTPSA id 4FMZyC5lsWz3x8J; Sat, 17 Apr 2021 03:20:19 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tugraz.at; s=mailrelay; t=1618622419; bh=nfPRcIUrN8LwewFkcTsK7gwajcJY4Ql+gct7XHj5iRk=; h=Subject:From:To:Date:In-Reply-To:References; b=Bs+0ok8/Lh4c1RO9IvZ1RwFhavW0E3dd45Z/1aYUhdz+pX5aHg1gHqw6w4J3aR/dn 05eCoRSXwztkQKjZGh72GBEs5w2Qn8XM/vY0NKwzZF2ZyenKQ4YNHJteECKtCWqMSW VrcAKWWt5AlV0q589M+XNvXVH6R/GRuLi9p7kFG4= Message-ID: <0ce900c16d59aa5a968c9ce92b4fdd3ebcc0e4fd.camel@student.tugraz.at> Subject: Re: Help adding a python package: chia-blockchain From: Leo Prikler To: Vladilen Kozin , guix-devel@gnu.org Date: Sat, 17 Apr 2021 03:20:17 +0200 In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.34.2 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-TUG-Backscatter-control: bt4lQm5Tva3SBgCuw0EnZw X-Spam-Scanner: SpamAssassin 3.003001 X-Spam-Score-relay: -1.9 X-Scanned-By: MIMEDefang 2.74 on 129.27.10.117 Received-SPF: pass client-ip=129.27.2.202; envelope-from=leo.prikler@student.tugraz.at; helo=mailrelay.tugraz.at X-Spam_score_int: -42 X-Spam_score: -4.3 X-Spam_bar: ---- X-Spam_report: (-4.3 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1618623616; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=nfPRcIUrN8LwewFkcTsK7gwajcJY4Ql+gct7XHj5iRk=; b=kX6pL75PUzTKSql1wixWE6vPH2oYq3L2ornuGkDq2T13b6Gan8gC8XhaDkd15ZBAPWWux0 68EH9S203nfCl7InlmrdQjLyOgv/lMgy34GjO5B1RMaoG6itE0PICwsFJiKTdlsS0OqIou hfF0FqJ3kvRmta7R2gI6yqXFRVKDkQ5/v29KyKWKBspImi22TiCGD8+L9RNku88g4N/Ipf ly6BQfTGET6t60Q35CIbPmNIGXzehxQGzvl+rz3kIqh0mOA2c1T0yToZEVHD/t1DiyhL1m KrswJCdzlyVXrcmt1GLHQdjwLS7L9Xb4EVlEkaodxWKH0ZaYBmxiZOByJoFAlQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1618623616; a=rsa-sha256; cv=none; b=nMi3EoV4BfbVdjb2y8oLBQKCevI8Tfwi0+0iS2DJVll7oK2YnPRpTgh8uUFBd/AETuaWZI 6jCYYWnLGW3+PacQS7L27/WU5qu/UqXOGo6zt6iq4wcHOs4RQPASIML7uAminXwJvKad2K Ss2RNWAwYbVFqjThiSV+rQd8yisr4rPpSJkOXbqdSh7rbgP9iSM0z7mejKb/ajm3j1J9nj gPpAfMuIoiDid77LquBGVNNheDvLLDEhYd/TmjGeBwopbK2+N6rN1CvbtJOK1umErHM3BL f3VU+PePFLvyJyHFu/G083QZyvMLYqdeGCnW6YqIeCdbHBIJU0MTazDxK5kFzA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=tugraz.at header.s=mailrelay header.b="Bs+0ok8/"; dmarc=pass (policy=none) header.from=student.tugraz.at; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -3.14 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=tugraz.at header.s=mailrelay header.b="Bs+0ok8/"; dmarc=pass (policy=none) header.from=student.tugraz.at; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 486EB235C4 X-Spam-Score: -3.14 X-Migadu-Scanner: scn0.migadu.com X-TUID: ayNCE1AUGUFJ Hi Vladilen, Am Freitag, den 16.04.2021, 09:57 +0100 schrieb Vladilen Kozin: > Hi all. > > Could a kind soul handhold me while I try to port a python package to > guix please. I am not a python programmer, so not on first name basis > with python ecosystem and new to guix - double trouble. > > I've made some progress, but could use a bit of your help. Current > state of affairs is tracked in this repo: > https://github.com/vkz/ze-guix Some comments on what you've written so far: ;; wtf? surely we don't need that on Linux Probably not, yeah. If you want to, you can substitute* references to that package away. ;; can we just skip these since this is only for testing? Yes, but you'll have to disable tests :) Otherwise mostly LGTM. > Repo and package I'm trying to import: > https://github.com/Chia-Network/chia-blockchain > > Both it and all of its dependencies are open source, most on pypi but > sadly some are only available there as binary wheels. Building from > source entirely maybe complicated since it very quickly escalates to > multiple build systems e.g. chiapos (proof of space lib) and chiavdf > are C++ and rely on cmake, clvm_rs (their homegrown lisp) is written > in Rust etc. They have thought about writing a Lisp implementation in Rust, but has anyone thought about writing a Rust compiler in Lisp? Perhaps it could even output shared objects, that'd be nice. Jokes aside, let's continue. > Conceptually, installation is very simple and basically relies on > binary wheels and amounts to: > - checkout repo, > - init and update submodule(s) - really only mozilla-ca, > - create virtual env and activate, > - upgrade to the latest pip to use binary wheels, > - install deps and the package with pip > > You can see the steps I extracted from their install.sh here: > https://github.com/vkz/ze-guix/blob/master/packages/manual-install.log > > Now, guix obviously heads and shoulders above virtualenv or at least > it should be, however trying to build from source escalates quickly > as > I've mentioned. So, first question is if I could define a package by > "replicating" the above essentially binary-only steps but replace > virtualenv with e.g. guix profile or smth. This should be possible to > do in code - as a proper package with some rudimentary build system > that amounts to building everything inside a separate profile or > maybe > just pulling binary wheels into a directory in store and then adding > that to whatever Python or pip expect and use as their PATH > equivalent. IMO doing something trivial like this should be easy. To > be sure this approach has security and "freedom" implications, but > this is orthogonal to having something that works installed quickly. > Building from source is very often involved borderline crazy. Could > someone kindly teach me how to do this "binary" only packaging. We have copy-build-system and a tool called patchelf. Some folks, who don't care as much about freedom as we do might even go ahead and implement a patchelf-build-system, but that's a little besides the point. If you feel like fiddling around in those binaries will be quicker than mixing build systems, that's pretty much the approach you would take, but it's not an easy one. > Now, to building from source. > > First step I tried was to `guix import pypi -r`. This fails because > not all deps have source pushed to pypi and only offer binary wheels. > IMO `guix import` has no business failing like this or even at all > for > whatever reason. It is best effort tool to begin with that should > result in a template that I can fill in. If there are errors - skip > the dependency, report it, but please continue. I don't think this works as a general statement. While UX probably could be improved in certain scenarios, sometimes an error really is an error and stopping is the right decision. > Since, recursive `guix import` failed me I had to manually track deps > and essentially call `guix import` for each one. Result you can see > here: > https://github.com/vkz/ze-guix/blob/master/packages/chia.scm > > This won't build for several reasons. Not all deps are there, some > probably shouldn't be like win32 something or other. Another reason, > I'd have to figure out how to build the inputs that require cmake and > rust and python (some all at once). And this is where I could really > use help. Examples as mentioned earlier are these dependencies: > https://github.com/Chia-Network/chiapos > https://github.com/Chia-Network/clvm_rs > maybe others - it is a fairly big project and I've not even touched > NPM/node that they rely on for their GUI client. That's another scary > can of worms I'm not prepared to peek inside at this time. This indeed sounds like a very daunting project. Rest assured, even the most advanced packagers usually shy away from such beasts. (And those who don't typically sacrifice part of their sanity.) Now in principle, mixing build systems is *easy*. Starting from (arguments), you add: - #:modules, that list both build systems, usually prefixed (e.g. gnu: and cmake:) - #:imported-modules, that hold the module closure for both (this can be achieved by listing ,@%gnu-build-system-modules ,@%cmake-build- system-modules and some others, for example) - #:phases, that call the right procedures in the correct order. A fairly simple package, that uses two build systems, is emacs-howm. After wip-emacs is merged, you have even more examples of emacs packages mixing build systems for trivial stuff, since that's how I chose to implement ELPA directories. I'm getting sidetracked again. Once you start seeing this pattern, you'll notice it in other packages as well. To search for packages, that use cmake in combination with something else, grepping for cmake: is a good idea. Likewise with cargo: for Rust packages. > Ok, so really two questions: > - how can I build a "binary" only python package that simply relies > on > binary wheels e.g. from pypi, By using a clever mix of "cp" and "patchelf" or their lisp equivalents. > - how to deal with multi-build systems e.g. python + cmake or python > + > rust, maybe someone could think of a good educational (so, not too > convoluted) example of such package already in source tree. You can't get much shorter than emacs-howm. > Latter would probably go much faster and teach better if done as > pair-programming exercise, but that maybe a lot to ask. If you're up > for it, I'm game though. I'm sorry, it's rather late and I should probably go to bed, but as far as contributing packages with huge dependency chains goes, there seem to be somewhat regular meetups done; perhaps someone else has more appropriate info for you. Lastly, if the dependencies themselves have a use other than being a dependency for what you're trying to package, you could garner some interest by sending one of them, that already builds fine, to guix-patches. Though perhaps that "if" is a little too big to be applicable here. In any case, have fun mixing build systems and don't get too discouraged by the huge load of work that's ahead of you. Regards, Leo