Packaging a python package that is using setuptools_rust

Packaging a python package that is using setuptools_rust

[Hamzeh Nasajpour]

Hi,

I want to package the `etebase-py` library: https://github.com/etesync/etebase-py/

It's using the `setuptools_rust` for build/install. Also it has some rust dependencies that I packaged them. And now I create a package definition by `guix import ...`:
```
(define-public python-etebase
  (package
    (name "python-etebase")
    (version "0.30.0")
    (source
      (origin
        (method url-fetch)
        (uri (pypi-uri "etebase" version))
        (sha256
          (base32
            "1py635aqnxx3jy4x6zffriqi0l9gc2gk06h0ms91k4rs68h0gb0i"))))
    (build-system python-build-system)
    (native-inputs
      `(("python-msgpack" ,python-msgpack)
        ("python-setuptools-rust" ,python-setuptools-rust)
        ("rust" ,rust)))
    (home-page
      "https://github.com/etesync/etebase-py")
    (synopsis "Python client library for Etebase")
    (description "Python client library for Etebase")
    (license #f)))
```

I faced with the following error in the package building:
```
...
etebase-0.30.0/setup.py
etebase-0.30.0/src/
etebase-0.30.0/src/lib.rs
etebase-0.30.0/src/glue.rs.in
phase `unpack' succeeded after 0.0 seconds
starting phase `ensure-no-mtimes-pre-1980'                                                                                                                                                                                                  
phase `ensure-no-mtimes-pre-1980' succeeded after 0.0 seconds
starting phase `enable-bytecode-determinism'                                                                                                                                                                                                
phase `enable-bytecode-determinism' succeeded after 0.0 seconds
starting phase `patch-usr-bin-file'                                                                                                                                                                                                         
phase `patch-usr-bin-file' succeeded after 0.0 seconds
starting phase `patch-source-shebangs'                                                                                                                                                                                                      
phase `patch-source-shebangs' succeeded after 0.0 seconds
starting phase `patch-generated-file-shebangs'                                                                                                                                                                                              
phase `patch-generated-file-shebangs' succeeded after 0.0 seconds
starting phase `build'                                                                                                                                                                                                                      
running "python setup.py" with command "build" and parameters ()
running build
running build_py
creating build
creating build/lib
creating build/lib/etebase
copying etebase/__init__.py -> build/lib/etebase
running build_ext
running build_rust
error: [Errno 2] No such file or directory: 'cargo'
command "python" "-c" "import setuptools, tokenize;__file__='setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\\r\\n', '\\n');f.close();exec(compile(code, __file__, 'exec'))" "build" failed with status 1
builder for `/gnu/store/lnwxp7sym6i2ixg4mnmz9h7ff1fg32cd-python-etebase-0.30.0.drv' failed with exit code 1
build of /gnu/store/lnwxp7sym6i2ixg4mnmz9h7ff1fg32cd-python-etebase-0.30.0.drv failed
View build log at '/var/log/guix/drvs/ln/wxp7sym6i2ixg4mnmz9h7ff1fg32cd-python-etebase-0.30.0.drv.bz2'.
guix build: error: build of `/gnu/store/lnwxp7sym6i2ixg4mnmz9h7ff1fg32cd-python-etebase-0.30.0.drv' failed
```

Do you have any idea? I should add the rust dependencies to `native-inputs` or `inputs` of `etebase-py`? Do you have any package definition that is used `setuptools_rust` as example for me?


Regards,
Hamzeh

Re: Packaging a python package that is using setuptools_rust

[Hartmut Goebel]

Am 19.10.20 um 12:52 schrieb Hamzeh Nasajpour:
> running build_rust
> error: [Errno 2] No such file or directory: 'cargo'
> command "python" "-c" "import setuptools, tokenize;__file__='setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\\r\\n', '\\n');f.close();exec(compile(code, __file__, 'exec'))" "build" failed with status 1
This looks like cargo is not found on the path, although it should be
there. You could validate $PATH like this: Add a phase before the
build-phase, containing just "(pk (getenv PATH))".

> Do you have any idea? I should add the rust dependencies to `native-inputs` or `inputs` of `etebase-py`?

It must be a native-input, since cargo and rustc are expected to run on
the build system.


-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Packaging a python package that is using setuptools_rust

[Hamzeh Nasajpour]

Hi Hartmut,

> You could validate $PATH like this: Add a phase before the
> build-phase, containing just "(pk (getenv PATH))".

This is the output of this line:
```
/gnu/store/wc4xh9wj49rlbyivxi4lsws6whhzx4ab-python-wrapper-3.8.2/bin:/gnu/store/axjq514q8gqk90hi7r7wpfbba3id6h18-rust-1.39.0/bin:/gnu/store/v6f44zccwh9z5zk3pjlywjybbi8n2hjh-tar-1.32/bin:/gnu/store/ncydgq2znms5n1d2k5yqshhf58nsixwv-gzip-1.10/bin:/gnu/store/i8h2pcxqdq07ijm3ibkka8f4smn1w48v-bzip2-1.0.8/bin:/gnu/store/9860f1abqj8wjjnwl8a9v54pdcc3bhgf-xz-5.2.4/bin:/gnu/store/60g7r3l01fd7c58yjbm6krgcwj1jkpwg-file-5.38/bin:/gnu/store/n4n560pfvvw50a9369axw5vj5rrqfj1n-diffutils-3.7/bin:/gnu/store/cd5qf3kcnlq35p9k392pjdpdzpsnds70-patch-2.7.6/bin:/gnu/store/hic7snhayfl7m6cpfqqr73nmm19bpqkg-findutils-4.7.0/bin:/gnu/store/swqdvwri9dbv6zssg6v0by7l05hd6wxp-gawk-5.0.1/bin:/gnu/store/ishk7fswcs4gkwcp8mh788z4mvvl9bxh-sed-4.8/bin:/gnu/store/bhs4rj58v8j1narb2454raan2ps38xd8-grep-3.4/bin:/gnu/store/57xj5gcy1jbl9ai2lnrqnpr0dald9i65-coreutils-8.32/bin:/gnu/store/hm40bxnv8jxmbc1lpb7zfimii4xm9m81-make-4.3/bin:/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16/bin:/gnu/store/mpa04aq8lblbcviyxywxcsb1zbi0mf39-ld-wrapper-0/bin:/gnu/store/m1z7cdbqsqyp9xnjw5cvlb4a7gkcg3m4-binutils-2.34/bin:/gnu/store/rn75fm7adgx3pw5j8pg3bczfqq1y17lk-gcc-7.5.0/bin:/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/bin:/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/sbin:/gnu/store/q9rm8h9imazsq2c4qiv2yjpvlvliywqb-python-3.8.2/bin
```

> /gnu/store/axjq514q8gqk90hi7r7wpfbba3id6h18-rust-1.39.0/bin

As you can see, the `rust` path is available in the `PATH`. I get the list of this path:
```
$ ls /gnu/store/axjq514q8gqk90hi7r7wpfbba3id6h18-rust-1.39.0/bin/
rustc  rustdoc  rust-gdb  rust-gdbgui  rust-lldb
```

I couldn't see the `cargo`. I also couldn't find any package for installing the `cargo`.


Regards,
Hamzeh

Re: Packaging a python package that is using setuptools_rust

[Tobias Geerinckx-Rice]

[-- Attachment #1: Type: text/plain, Size: 772 bytes --]

Hamzeh,

Hamzeh Nasajpour 写道：
> As you can see, the `rust` path is available in the `PATH`. I 
> get the list of this path:
> ```
> $ ls 
> /gnu/store/axjq514q8gqk90hi7r7wpfbba3id6h18-rust-1.39.0/bin/
> rustc  rustdoc  rust-gdb  rust-gdbgui  rust-lldb
> ```
>
> I couldn't see the `cargo`. I also couldn't find any package for 
> installing the `cargo`.

You're missing the rust:cargo output that provides ‘cargo’:

  ~ λ guix build rust
  ...
  /gnu/store/a5dmqx64mg8jq5s9ffrp4vwhmw3wjjdg-rust-1.46.0-cargo
  /gnu/store/4szzm4wlbrpspc7xi8vqx4hckazqn78i-rust-1.46.0-doc
  /gnu/store/hq4ni80kwx1cj6vyx35ljzr092xgvxca-rust-1.46.0

I.e.:

  (native-inputs
   `(...
     ("rust:cargo" ,rust "cargo")
     ...))

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: Packaging a python package that is using setuptools_rust

[Hamzeh Nasajpour]

Thanks for your quick reply.


>   (native-inputs
>    `(...
>      ("rust:cargo" ,rust "cargo")
>      ...))

Thanks this step passed, but now I face with another error in dependencies. 

```
...
phase `patch-generated-file-shebangs' succeeded after 0.0 seconds
starting phase `build'                                                                                                                                                                                                                      
running "python setup.py" with command "build" and parameters ()
running build
running build_py
creating build
creating build/lib
creating build/lib/etebase
copying etebase/__init__.py -> build/lib/etebase
running build_ext
running build_rust
error: failed to load source for a dependency on `cpython`

Caused by:
  Unable to update registry `https://github.com/rust-lang/crates.io-index`

Caused by:
  failed to create directory `/homeless-shelter/.cargo/registry/index/github.com-1ecc6299db9ec823`

Caused by:
  Permission denied (os error 13)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "setup.py", line 4, in <module>
...
```

indeed, this library is python and internally it's using a rust library. the `cpython` is one its dependencies that I packaged it (`rust-cpython`) and it needs to it. I don't know how can I add rust  dependencies here since I'm using python build system. I added it to inputs but not resolved.

Indeed now the problem is adding dependencies (some of them rust and some of them python) to this package while I'm using the python build system. Do you have any idea?


Regards,
Hamzeh

Re: Packaging a python package that is using setuptools_rust

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2547 bytes --]

On Sun, Oct 25, 2020 at 02:58:22PM +0330, Hamzeh Nasajpour wrote:
> Thanks for your quick reply.
> 
> 
> >   (native-inputs
> >    `(...
> >      ("rust:cargo" ,rust "cargo")
> >      ...))
> 
> Thanks this step passed, but now I face with another error in dependencies. 
> 
> ```
> ...
> phase `patch-generated-file-shebangs' succeeded after 0.0 seconds
> starting phase `build'                                                                                                                                                                                                                      
> running "python setup.py" with command "build" and parameters ()
> running build
> running build_py
> creating build
> creating build/lib
> creating build/lib/etebase
> copying etebase/__init__.py -> build/lib/etebase
> running build_ext
> running build_rust
> error: failed to load source for a dependency on `cpython`
> 
> Caused by:
>   Unable to update registry `https://github.com/rust-lang/crates.io-index`
> 
> Caused by:
>   failed to create directory `/homeless-shelter/.cargo/registry/index/github.com-1ecc6299db9ec823`
> 
> Caused by:
>   Permission denied (os error 13)
> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
>   File "setup.py", line 4, in <module>
> ...
> ```
> 
> indeed, this library is python and internally it's using a rust library. the `cpython` is one its dependencies that I packaged it (`rust-cpython`) and it needs to it. I don't know how can I add rust  dependencies here since I'm using python build system. I added it to inputs but not resolved.
> 
> Indeed now the problem is adding dependencies (some of them rust and some of them python) to this package while I'm using the python build system. Do you have any idea?
> 

This one is a little harder. You can add cargo-build-system phases into
the python-build-system (see icecat for an example) and add a phase to
place the rust dependencies _sources_ in the correct location, or you
can switch it over to a cargo-build-system and add back the
python-build-system phases (see newsboat for an example). IMO the
cargo-build-system option is easier since it takes care of dependency
management for you in terms of #:cargo-inputs.

> 
> Regards,
> Hamzeh
> 
> 
> 

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Packaging a python package that is using setuptools_rust

[Lars-Dominik Braun]

Hi Hamzeh,

> I want to package the `etebase-py` library: https://github.com/etesync/etebase-py/
did you manage to package this library? I’m also interested in it,
because I want to try etesync-dav.

Thanks,
Lars

Re: Packaging a python package that is using setuptools_rust

[phodina]

Hi Hamzeh and Lars,

> Hi Hamzeh,
>
> > I want to package the `etebase-py` library: https://github.com/etesync/etebase-py/
>
> did you manage to package this library? I’m also interested in it,
>
> because I want to try etesync-dav.
>
> Thanks,
>
> Lars

I'm also interested in packaging this Python module as Rust makes it's way into Python universe.

Also since it's setuptools should we also modify the Python build system?

----
Petr