Re: Help adding a python package: chia-blockchain

[Vladilen Kozin <vladilen.kozin@gmail.com>]

Another "hilarious" consequence of attempting to side-step "the guix"
way. If I just follow the `pip install` steps I end up with some C++
dependency (blspy I believe) complaining that it can't find libstdc++.
Quick `locate` search unsurfaced it somewhere under /gnu/store, but of
course binaries (like binary wheels) pre-built this way are completely
oblivious to Linux machines (or I suppose Guix) that don't have the
usual /lib or /lib64 or whatever. So yeah, looks like its guix way or
no way. Like I mentioned earlier, could use some guidance in packaging
this.

On Fri, 16 Apr 2021 at 09:57, Vladilen Kozin <vladilen.kozin@gmail.com> wrote:
>
> Hi all.
>
> Could a kind soul handhold me while I try to port a python package to
> guix please. I am not a python programmer, so not on first name basis
> with python ecosystem and new to guix - double trouble.
>
> I've made some progress, but could use a bit of your help. Current
> state of affairs is tracked in this repo:
> https://github.com/vkz/ze-guix
>
> Repo and package I'm trying to import:
> https://github.com/Chia-Network/chia-blockchain
>
> Both it and all of its dependencies are open source, most on pypi but
> sadly some are only available there as binary wheels. Building from
> source entirely maybe complicated since it very quickly escalates to
> multiple build systems e.g. chiapos (proof of space lib) and chiavdf
> are C++ and rely on cmake, clvm_rs (their homegrown lisp) is written
> in Rust etc.
>
> Conceptually, installation is very simple and basically relies on
> binary wheels and amounts to:
> - checkout repo,
> - init and update submodule(s) - really only mozilla-ca,
> - create virtual env and activate,
> - upgrade to the latest pip to use binary wheels,
> - install deps and the package with pip
>
> You can see the steps I extracted from their install.sh here:
> https://github.com/vkz/ze-guix/blob/master/packages/manual-install.log
>
> Now, guix obviously heads and shoulders above virtualenv or at least
> it should be, however trying to build from source escalates quickly as
> I've mentioned. So, first question is if I could define a package by
> "replicating" the above essentially binary-only steps but replace
> virtualenv with e.g. guix profile or smth. This should be possible to
> do in code - as a proper package with some rudimentary build system
> that amounts to building everything inside a separate profile or maybe
> just pulling binary wheels into a directory in store and then adding
> that to whatever Python or pip expect and use as their PATH
> equivalent. IMO doing something trivial like this should be easy. To
> be sure this approach has security and "freedom" implications, but
> this is orthogonal to having something that works installed quickly.
> Building from source is very often involved borderline crazy. Could
> someone kindly teach me how to do this "binary" only packaging.
>
> Now, to building from source.
>
> First step I tried was to `guix import pypi -r`. This fails because
> not all deps have source pushed to pypi and only offer binary wheels.
> IMO `guix import` has no business failing like this or even at all for
> whatever reason. It is best effort tool to begin with that should
> result in a template that I can fill in. If there are errors - skip
> the dependency, report it, but please continue.
>
> Since, recursive `guix import` failed me I had to manually track deps
> and essentially call `guix import` for each one. Result you can see
> here:
> https://github.com/vkz/ze-guix/blob/master/packages/chia.scm
>
> This won't build for several reasons. Not all deps are there, some
> probably shouldn't be like win32 something or other. Another reason,
> I'd have to figure out how to build the inputs that require cmake and
> rust and python (some all at once). And this is where I could really
> use help. Examples as mentioned earlier are these dependencies:
> https://github.com/Chia-Network/chiapos
> https://github.com/Chia-Network/clvm_rs
> maybe others - it is a fairly big project and I've not even touched
> NPM/node that they rely on for their GUI client. That's another scary
> can of worms I'm not prepared to peek inside at this time.
>
> Ok, so really two questions:
> - how can I build a "binary" only python package that simply relies on
> binary wheels e.g. from pypi,
> - how to deal with multi-build systems e.g. python + cmake or python +
> rust, maybe someone could think of a good educational (so, not too
> convoluted) example of such package already in source tree.
>
> Latter would probably go much faster and teach better if done as
> pair-programming exercise, but that maybe a lot to ask. If you're up
> for it, I'm game though.
>
> Thank you
> --
> Best regards
> Vlad Kozin



-- 
Best regards
Vlad Kozin