rust (build system) deficits

rust (build system) deficits

[Hartmut Goebel]

Hi,

I just package "nettle-sys", which are binding for the nettle crypto
library. Builiding these bindings takes quite some time, some native
inputs and - worst of all - some special phases. After building, the
output is empty (beside some license files). Now when building `nettle`,
which depends on `nettle-sys`, I - as expected - get errors, since e.g.
`bindgen` is missing.

While I could work-around this issue for this case, IMO this shows a
much, much deeper and serious problem with rust and the rust
build-system: We can not pre-built packages!

Rust not being able toe pre-build packages is an open issue, see
<https://github.com/rust-lang/cargo/issues/1139>. But this issue also
points out two things:

1. There seems to be some `.rlib` pre-built format. And "This is a Rust
problem even more than a Cargo problem. You can't guarantee that a
pre-built Rust library will work unless it's built with the exact same
SHA of the compiler." OTOH sis means: since we can guarantee the very
same compiler to be used, we could use rust libraries.

2. There is a tool/library DHL <https://github.com/Popog/dhl/>
(Dependency Hijacking Library) which tries to solve this issue. Maybe
this helps.

If anybody is working on a solution: Thanks :-)

Side note: This also is an ecological problem, as each build will
consume more power and CO2 then required. This is esp. true for guix
(and nix), where packages need to be rebuild much more often than on
non-functional distributions.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: rust (build system) deficits

[John Soo]

Hi Hartmut,

I agree with you. It seems like a problem for cargo to solve to me.  Efraim tried to use the .rlib files to make library files and found it was not really an option.

There are more problems, too. The way inputs are done doesn’t fit well with the rest of guix tooling and doesn’t really follow functional package management concepts.

One possibility to try is to rethink the cargo-{development-}inputs fields and only require the source from rust library inputs. Then to build the executables, it would only require one compile and we could keep ci checking the current libraries.

The cargo build system also has a specialized transitive closure computation. I believe the reason cargo-{development-}inputs are specified in arguments is for the special purpose of that closure computation. Can someone tell me if I’m wrong on that?

As far as I can tell, the point of the specialized transitive closure computation is to deal with cyclic dependencies.  Again, please someone correct me if I’m wrong on that.

If the rust closure functions successfully deal with cyclic dependencies, then wouldn’t the other closure computations benefit from the same function?

I would like to take a stab at bringing the rust build system package definitions closer to others.

My proposal is to:
* move cargo-{development-}inputs into inputs, requiring only the source from libraries.
* either:
  - move the rust closure function so all packages use it
Or
  - adjust the transitive closure function such that it works on normal inputs rather than arguments
* Do not build rust packages by default. Only run tests.
* as a corollary to the previous item: Default skip-build? to #f but do run tests even if skip-build is #f and tests? is #t

What do you all think, Hartmut, guix?

- John 

Re: rust (build system) deficits

[Hartmut Goebel]

Hi John,
> My proposal is to:
> * move cargo-{development-}inputs into inputs, requiring only the source from libraries.
> * either:
>   - move the rust closure function so all packages use it
> Or
>   - adjust the transitive closure function such that it works on normal inputs rather than arguments
> * Do not build rust packages by default. Only run tests.
> * as a corollary to the previous item: Default skip-build? to #f but do run tests even if skip-build is #f and tests? is #t

Actually I do not care so much about  cargo-{development-}inputs fields.

The much more serious issue is that we are not able to build non-trivial
Rust applications: Given a package which needs to add phases, e.g. for
fixing Cargo.toml, we would need to run each package's phases when
building any depending package.


-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: rust (build system) deficits

[John Soo]

Hi Hartmut,

> On Mar 8, 2020, at 10:16 AM, Hartmut Goebel <h.goebel@crazy-compilers.com> wrote:

The much more serious issue is that we are not able to build non-trivial
Rust applications: Given a package which needs to add phases, e.g. for
fixing Cargo.toml, we would need to run each package's phases when
building any depending package.

Hmm. Can you elaborate more on “not able to build non-trivial rust applications”?  It seems like we have finally made some progress on building some rust apps.  Patching a library Cargo.toml seems like an excellent job for source patches or snippets. Modifying an executable’s Cargo.toml to refer to guix-vendor happens during a phase that matters (because the executable will be built).

We also already have solutions for removing censored code for c library wrappers.

Clearly the situation is not ideal, but I do think there are improvements to take just with what we have now.

Kindly,

John

Re: rust (build system) deficits

[John Soo]

Also I should just say thank you for taking interest in the cargo-build system. Please let me know if I’m missing something.

John

Re: rust (build system) deficits

[Hartmut Goebel]

Am 08.03.20 um 21:10 schrieb John Soo:
> Hmm. Can you elaborate more on “not able to build non-trivial rust applications”?

I'm working on packaging sequoia-openpgp.org.

This requires nettle-sys, a FFI to nettle. nettly-sys "optionally"
requires "nettle-src". I added a phase to "nettle-sys" to remove the
"optional" dependency. nettle-sys also requires bindgen.

A second dependancy is "sequoia-openpgp", which requires rhe lalrpop
parser generator for building.

Now when building `sequioa-sqv`, I need to add all these dependencies again:

- nettle-src, since it is "optional" for nettle-sys - and the phase was
not executed.
- bindgen, since it is required to build nettle-sys
- lalrpop, since it is required for building sequoia-openpgp

> Patching a library Cargo.toml seems like an excellent job for source patches or snippets.

Quite some packages change Cargo.toml in a phase, e.g. rust-openssl-0.7.
Thus I assumed, this is the way to go.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: rust (build system) deficits

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1815 bytes --]

On Mon, Mar 09, 2020 at 10:26:01AM +0100, Hartmut Goebel wrote:
> Am 08.03.20 um 21:10 schrieb John Soo:
> > Hmm. Can you elaborate more on “not able to build non-trivial rust applications”?
> 
> I'm working on packaging sequoia-openpgp.org.
> 
> This requires nettle-sys, a FFI to nettle. nettly-sys "optionally"
> requires "nettle-src". I added a phase to "nettle-sys" to remove the
> "optional" dependency. nettle-sys also requires bindgen.

As I assume you saw, there is nothing "optional" when using the
cargo-build-system. For nettle-sys and nettle-src I suggest looking at
rust-openssl-sys-no-vendor.patch which I modified from Debian. It takes
care of pretending openssl-sys also provides openssl-src.

> A second dependancy is "sequoia-openpgp", which requires rhe lalrpop
> parser generator for building.
> 
> Now when building `sequioa-sqv`, I need to add all these dependencies again:
> 
> - nettle-src, since it is "optional" for nettle-sys - and the phase was
> not executed.
> - bindgen, since it is required to build nettle-sys
> - lalrpop, since it is required for building sequoia-openpgp
> 
> > Patching a library Cargo.toml seems like an excellent job for source patches or snippets.
> 
> Quite some packages change Cargo.toml in a phase, e.g. rust-openssl-0.7.
> Thus I assumed, this is the way to go.

That could probably move to a snippet, or even into the
cargo-build-system. IIRC there are only 3 patterns like that which
appear multiple times, and they're only relevant when building that
actual crate, not when it's "just a dependency".

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: rust (build system) deficits

[John Soo]

Hi Hartmut,

> On Mar 9, 2020, at 2:26 AM, Hartmut Goebel <h.goebel@crazy-compilers.com> wrote:
> 

A second dependancy is "sequoia-openpgp", which requires rhe lalrpop
parser generator for building.

Now when building `sequioa-sqv`, I need to add all these dependencies again:

- nettle-src, since it is "optional" for nettle-sys - and the phase was
not executed.
- bindgen, since it is required to build nettle-sys
- lalrpop, since it is required for building sequoia-openpgp

That is a lot of work! I know because I did all of those for pijul in this channel github.com/jsoo1/guix-channel.  Want to compare notes? What are you working on? 

Also I found the recursive crate importer very helpful, even if it was imperfect. Are you using the importer or doing the work by hand?

And as Efraim mentioned, optional dependencies in the cargo sense are not optional for guix definitions.

Good luck,

John

Re: rust (build system) deficits

[Hartmut Goebel]

Am 09.03.20 um 15:00 schrieb John Soo:

>> - nettle-src, since it is "optional" for nettle-sys - and the phase was
>> not executed.
>> - bindgen, since it is required to build nettle-sys
>> - lalrpop, since it is required for building sequoia-openpgp
>
> That is a lot of work! […] Also I found the recursive crate importer very helpful, even if it was imperfect. Are you using the importer or doing the work by hand?

My point is less the work, but the non-transitive declarations:
nettle-sys is an multi-indirect input for sequioa-sqv, still the later
needs to specify these dependencies.

This importer does not solve the declarations, and IMHO it should not
anyway - as the are dependencies of another packages, which might change
over time.

> Want to compare notes? What are you working on? 
Sure: <https://gitlab.digitalcourage.de/htgoebel/guix/-/tree/HG-squoia>

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: rust (build system) deficits

[John Soo]

Hi Hartmut,

> My point is less the work, but the non-transitive declarations:
nettle-sys is an multi-indirect input for sequioa-sqv, still the later
needs to specify these dependencies.

Totally agree. I think everyone agreed, too.  A few months ago we decided that the package inputs should match as close to exactly the dependencies listed in Cargo.toml so as to avoid specifying transitive dependencies in the package definition. 

The importer does solve the transitive dependencies but there is a bug. Version numbers of cargo dependencies are not used which can sometimes cause the problem you describe. I really hope the fixes get merged soon because it is a real pain. 

> This importer does not solve the declarations, and IMHO it should not
anyway - as the are dependencies of another packages, which might change over time.

I’m not sure I fully understand why the recursive importer should not solve the transitive dependencies. Could you elaborate further?

If you are suggesting that guix refresh won’t pick up the changes, then I think agree with you. That I believe is an artifact of using arguments rather than inputs to specify dependencies.

Kindly,

John

Re: rust (build system) deficits

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 1259 bytes --]

Hi John,
>> This importer does not solve the declarations, and IMHO it should not
>> anyway - as the are dependencies of another packages, which might change over time.
>>
> I’m not sure I fully understand why the recursive importer should not
> solve the transitive dependencies. Could you elaborate further?
>
> If you are suggesting that guix refresh won’t pick up the changes,
> then I think agree with you. That I believe is an artifact of using
> arguments rather than inputs to specify dependencies.
>
I was talking about `guix import crates -r`, which - like `guix refresh`
- shall not add transitive dependencies to #crate-inputs.

But I would expect the build-system to transitivly resolve #crate-inputs
as well as #crate-development inputs.

> The importer does solve the transitive dependencies but there is a bug. Version numbers of cargo dependencies are not used which can sometimes cause the problem you describe. I really hope the fixes get merged soon because it is a real pain. 
Are you talking about "guix import crates" here, too?

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |


[-- Attachment #2: Type: text/html, Size: 2121 bytes --]